Overview
- Learn more about the Spark architecture
- Learn about the different execution modes
Introduction
Apache SparkApache Spark is an open-source data processing engine that enables the analysis of large volumes of information quickly and efficiently. Its design is based on memory, which optimizes performance compared to other batch processing tools. Spark is widely used in big data applications, Machine Learning and Real-Time Analytics, thanks to its ease of use and... es un motor informático unificado y un conjunto de bibliotecas para el procesamiento de datos en paralelo en clústeres de computadoras. It is the most developed open source engine for this task, making it a standard tool for any developer or data scientist interested in big data.
Spark supports multiple widely used programming languages (Python, Java, Scale y R), includes libraries for various tasks ranging from SQL to streaming to machine learning, and Spark runs anywhere, desde una computadora portátil hasta un clusterA cluster is a set of interconnected companies and organizations that operate in the same sector or geographical area, and that collaborate to improve their competitiveness. These groupings allow for the sharing of resources, Knowledge and technologies, fostering innovation and economic growth. Clusters can span a variety of industries, from technology to agriculture, and are fundamental for regional development and job creation.... de cientos de servidores. This makes it an easy system to get started and scale up to big data processing or incredibly large scale..
More than 500 collaborators of 200 institutions responsible for the code and a user base of more than 225,000 members, Apache Spark has become the most common and requested big data framework across all major industries. Ecommerce companies like Alibaba, social media companies like Tencent and Chinese search engine Baidu run apache spark operations on a large scale.
This post is a one-time resource that provides an overview of the Spark architecture with the help of a Spark architecture diagram..
Table of Contents
- The architecture of a Spark application
- Spark's driver
- The Spark Enforcers
- The Cluster Administrator
- Types of Cluster Manager
- Execution modes
- Cluster mode
- Client way
- local mode
The architecture of a Spark application
Below are the high-level components of the Apache Spark application architecture:
Spark's driver
The driver is the procedure “in the driver's seat” from your Spark application. It is the controller of the execution of a Spark application and maintains all the states of the Spark cluster (the status and tasks of the executors). You must interact with the cluster administrator to obtain physical resources and run executors.
At the end of the day, this is just a procedure on a physical machine that is responsible for maintaining the state of the application running on the cluster.
The Spark Enforcers
Spark executors are the processes that perform the tasks assigned by the Spark controller. The executors have a fundamental responsibility: take the tasks assigned by the driver, run them and report on their status (success or failure) and results. Each Spark application has its own independent executing processes.
The cluster administrator
Spark Driver and Executors don't exist in a vacuum, and this is where the cluster manager comes in. The cluster administrator is responsible for maintaining a cluster of machines that will run your Spark applications. In a somewhat confusing way, a cluster administrator will have their own abstractions of “controller” (sometimes called teacher) Y “worker”.
The main difference is that they are linked to physical machines rather than processes (as they are in Spark). La máquina a la izquierda de la ilustración es el nodeNodo is a digital platform that facilitates the connection between professionals and companies in search of talent. Through an intuitive system, allows users to create profiles, share experiences and access job opportunities. Its focus on collaboration and networking makes Nodo a valuable tool for those who want to expand their professional network and find projects that align with their skills and goals.... de controlador de Cluster Manager. The circles represent daemon processes that are run and managed by each of the individual worker nodes. No Spark application is running yet; these are just the cluster manager processes.
When it's time to run a Spark application, We request resources from the cluster administrator to run it. Depending on how our application is configured, this can include a place to run the Spark controller or it can be just resources for the executors of our Spark application. Throughout the course of the Spark application execution, the cluster administrator will be responsible for managing the underlying machines our application runs on.
There are several useful things to pay attention to about this architecture:
- Each application has its own executing processes, that remain active throughout the application and run multi-threaded tasks. This has the advantage of isolating the applications from each other., both on the programming side (each controller schedules its own tasks) as executor side (different application tasks run on different JVMs).
Despite this, it also means that the data cannot be shared between different Spark applications (instances of SparkContext) without writing them to an external storage system. - Spark is independent of the underlying cluster administrator. As long as you can get executing processes and they communicate with each other, it is relatively easy to run even in a cluster administrator that also supports other applications (as an example, Months / YARNYARN is a package manager for JavaScript that allows the efficient installation and management of dependencies in development projects. Powered by Facebook, It is characterized by its speed and security compared to other managers. YARN uses a cache system to optimize installations and provides a lock file to ensure consistency of dependency versions across different development environments....).
- The controller program must listen and accept incoming connections from its executors throughout its life (as an example, watch spark.driver.port in the network settings section). As such, controller program must be network addressable from worker nodes.
- Because the controller schedules tasks on the cluster, must run near worker nodes, preferably on the same local area network. If you want to send requests to the cluster remotely, it is better to open an RPC to the controller and have it send operations from nearby than to run a controller away from worker nodes.
Types of cluster administrators
At the moment, the system supports multiple cluster administrators:
- Be unique – a simple cluster manager included with Spark that makes it easy to set up a cluster.
- Apache Mesos – un administrador de clúster general que además puede ejecutar Hadoop MapReduceMapReduce is a programming model designed to efficiently process and generate large data sets. Powered by Google, This approach breaks down work into smaller tasks, which are distributed among multiple nodes in a cluster. Each node processes its part and then the results are combined. This method allows you to scale applications and handle massive volumes of information, being fundamental in the world of Big Data.... y aplicaciones de servicio.
- Hadoop HILO – the resource manager in Hadoop 2.
- Governors – an open source system to automate deployment, scaling and managing containerized applications.
There is a third party project (not supported by Spark project) to add support for Nomadic as a cluster administrator.
Execution modes
A run mode gives you the power to determine where the aforementioned resources are physically located when you run your application. You have three modes to select:
- Cluster mode
- Client way
- local mode
Cluster mode
Cluster mode is probably the most common way to run Spark applications. In cluster mode, a user submits a JAR, a Python script or a precompiled R script to a cluster administrator. After, the cluster administrator launches the controller procedure on a worker node within the cluster, at the same time of the executing processes. This means that the cluster administrator is responsible for maintaining all processes related to the Spark application..
Client way
Client mode is almost the same as cluster mode, except that the Spark driver remains on the client machine that sent the request. This means that the client machine is responsible for maintaining the Spark controller procedure and the cluster administrator maintains the executing processes.. These machines are commonly known as link gate machines or edge nodes..
local mode
Local mode is a significant departure from the previous two modes: run the entire Spark application on a single machine. Achieve parallelism by means of threads on that single machine. This is a common way to learn Spark, test your applications or iteratively experiment with local development.
Despite this, we do not suggest using local mode to run production applications.
Conclution
In summary, Spark helps us break down heavy, compute-intensive jobs into smaller, more concise tasks that are then executed by worker nodes.. It also achieves real-time or archived data processing using its basic architecture..
I recommend that you check out the following data engineering resources to boost your knowledge:
I hope you liked the post. If you have any questions related to this post, let me know in the comment section below.
