In order to reduce the latency of big data processing operations and bring a series of improvements, Apache Software Foundation (ASF) has announced the availability of the first version of Spark, an open source analysis software that accelerates task execution real-time analysis about the Hadoop data processing platform.
Known as “the Hadoop Swiss Army Knife”, how this new tool works enriches the ecosystem from this distributed computing model that offers an interesting alternative to MapReduceMapReduce is a programming model designed to efficiently process and generate large data sets. Powered by Google, This approach breaks down work into smaller tasks, which are distributed among multiple nodes in a cluster. Each node processes its part and then the results are combined. This method allows you to scale applications and handle massive volumes of information, being fundamental in the world of Big Data..... Their replacement means benefits by allowing real-time analysis on Hadoop clusters and multiply speed by 100 in memory compared to MapReduce and up to 10 more times on disk.
Instead of running the jobs in batch mode, making cross-cluster real-time analysis impossible, Spark works in micro-batches intervals of five seconds or less, which also provides more stability than other real-time treatment tools.
Real-time analysis and ease of use
With this version 1.0 the Spark, Apache offers a stable application programming interface under Apache license 2.0, as with all the software that has emerged from the feverish activity of the foundation's incubator. further, Databriks distributes it in its commercial version.
For its compatibility, developers can also use it to enter MapReduce code in their own applications, as well as to create other written in Java, Scala or Python, three of the most frequent languages.
Be able to jointly analyze structured data together with other unstructured data in the same analysis operation and allow its use in small and large teams o CPDs are another of the most outstanding features of this version.
In addition to being compatible with the data sources of the file system HDFSHDFS, o Hadoop Distributed File System, It is a key infrastructure for storing large volumes of data. Designed to run on common hardware, HDFS enables data distribution across multiple nodes, ensuring high availability and fault tolerance. Its architecture is based on a master-slave model, where a master node manages the system and slave nodes store the data, facilitating the efficient processing of information.. (Hadoop’s Distributed File System), it is compatible with some of its components such as YARNYARN is a package manager for JavaScript that allows the efficient installation and management of dependencies in development projects. Powered by Facebook, It is characterized by its speed and security compared to other managers. YARN uses a cache system to optimize installations and provides a lock file to ensure consistency of dependency versions across different development environments.... (Yet Anoter Resource Netotiator) or with the databaseA database is an organized set of information that allows you to store, Manage and retrieve data efficiently. Used in various applications, from enterprise systems to online platforms, Databases can be relational or non-relational. Proper design is critical to optimizing performance and ensuring information integrity, thus facilitating informed decision-making in different contexts.... Distributed HBaseHBase is a NoSQL database designed to handle large volumes of data distributed in clusters. Based on the column model, Enables fast, scalable access to information. HBase easily integrates with Hadoop, making it a popular choice for applications that require massive data storage and processing. Its flexibility and ability to grow make it ideal for big data projects...., one of the Hadoop databases. .
A use oriented to the permanent analysis of data in real time is added other functionalities that revolve around its software library, among other graphical treatments or in-depth calculations involving machine learning, as well as interactive data queries.
The AMP laboratory (Algorithms, Machines and People) Berkeley initiated the creation of Spark, and in June 2013, a year ago, the The ASF community adopted the project to give you the maximum boost. Nowadays, Spark is in use in companies around the world, like IBM, Cloudera Intel or Pivotal have already integrated Spark into their Hadoop distributions, So there are high expectations that this new software will play an important role in Big Data data processing..
Created in 1999, The Foundation oversees dozens of open source projects and has contributed thousands of software solutions that are distributed under the Apache license., including the famous Apache HTTP server framework, the world's most popular distributed data processing system.
Related Post:
Image source: renjith krishnan / FreeDigitalPhotos.net
(function(d, s, id) {
var js, fjs = d.getElementsByTagName(s)[0];
if (d.getElementById(id)) return;
js = d.createElement(s); js.id = id;
js.src = “//connect.facebook.net/es_ES/all.js#xfbml=1&status=0”;
fjs.parentNode.insertBefore(js, fjs);
}(document, ‘script’, 'facebook-jssdk'));
