Hadoop is an open source framework for storing data and running applications on basic hardware clusters. Provides massive storage for any type of data, massive processing power and the ability to handle virtually unlimited tasks or jobs. This simply and in a very concrete way is what what is hadoop. Y, what's it for?
In some other post We have explained the history of Hadoop and how it was born from Google's need to be able to process all the data on the web. Let's now look at other important Hadoop concepts that will give us the keys to why Hadoop is essential, what are the challenges of using hadoop, how is it used, … in summary, What is Hadoop and what is it for?.
Why is Hadoop essential?
- Ability to quickly save and process large amounts of any type of data. With ever-increasing volumes and variety of data, especially when it comes to social media and the internet of things, this is a key consideration.
- Processing power. Hadoop's Distributed Computing Model Rapidly Processes Big Data. The more compute nodes you use, more processing power will have.
- Fault tolerance. Data and application processing is protected against hardware failure. If a node stops working, jobs are automatically redirected to other nodes to ensure distributed computing does not fail. Multiple copies of all data are automatically stored.
- Flexibility. Unlike traditional relational databases, no need to preprocess data before storing. You can store as much data as you want and choose how to use it later. This includes unstructured data such as text, images and video.
- Low cost. The open source framework is cost-free and uses basic hardware to store large amounts of data.
- Scalability. You can easily grow the system to handle more data by simply adding nodes. Little administration is needed.
What are the challenges of using Hadoop?
- Programar con MapReduceMapReduce is a programming model designed to efficiently process and generate large data sets. Powered by Google, This approach breaks down work into smaller tasks, which are distributed among multiple nodes in a cluster. Each node processes its part and then the results are combined. This method allows you to scale applications and handle massive volumes of information, being fundamental in the world of Big Data.... no es una buena opción para todos los problemas. Good for simple problems and information requests that can be divided into independent units, but not efficient for analytical tasks, iterative and interactive. MapReduce is file intensive and iterative algorithms require several phases of classification and map design to complete. This creates multiple files between MapReduce phases and is ineffective for advanced analytical computing.
- There is a widely recognized talent gap. It can be difficult to find level programmers who have enough Java knowledge to be productive with MapReduce. That's one of the reasons distribution providers compete to put relational SQL technology above Hadoop.. It is much easier to find programmers with SQL skills than with MapReduce skills. And managing Hadoop seems to be part art and part science, what you need a low level of knowledge of operating systems, Hadoop hardware and kernel configuration.
- Data security. Another challenge focuses on fragmented data security issues, even when new tools and technologies are emerging. The Kerberos authentication protocol is a great step to protect Hadoop environments.
- Administration and governance data. Hadoop does not have comprehensive, easy-to-use tools for data management, data cleaning, governance and metadata. It especially lacks tools for standardization and data quality.
How do you use Hadoop?
Beyond your original goal of searching millions or hundreds of millions of web pages and getting relevant results, what Hadoop is and what it is for is what many institutions look for in Hadoop. Companies are looking to make Hadoop their next great data platform. Today's most popular uses are:
- Low-cost data archiving and storage. Modest hardware cost makes Hadoop useful for storing and combining data as transactional, social media, sensors, machines, scientific data, etc. Low-cost storage enables you to keep information that is not currently considered critical but that you may need to analyze. after.
- Sandbox for discovery and analysis. Because Hadoop was designed to handle volumes of data in a variety of ways, can run analytical algorithms. The Big Data Analytics on Hadoop can help an organization operate more efficiently, discover new possibilities and gain a competitive advantage. The sandbox or sandbox approach offers a possibility to innovate with a minimal investment.
- Data lake. Data lakes enable data to be stored in its original or exact format, both structured and unstructured, and without any processing, in order to provide an unmodified or raw view of the data to data analysts so that they can use it. to discover and analyze. Helps them ask new or difficult questions without restriction. Data lakes are not a substitute for data warehouses. In reality, how to protect and control data lakes is a very important topic for IT.
- Complement your data warehouse. We are already seeing Hadoop coming alongside data warehousing environments, as well as certain data sets that are downloaded from the data warehouse to Hadoop, or new data types that go directly to Hadoop. The ultimate goal of each organization is to have a platform to store and process data from different schemes, formats, etc., to support different use cases that can be integrated at different levels.
- IoT y Hadoop. Things in IoT need to know what to communicate and when to act. At the core of IoT there is a constant flow of a torrent of data. Hadoop is often used as a data warehouse for millions or hundreds of millions of transactions. Mass storage and processing capabilities also allow you to use Hadoop as a discovery and pattern definition sandbox to be monitored for prescriptive instructions.. You can continually improve these instructions below, since Hadoop is constantly being used with new data that doesn't match previously defined patterns.
Conclution
We have seen What is Hadoop and what is it for? at the same time of the relevance it has at this time for companies and the challenges of using it due to some complication to find experts in the field. Now you can start using it to get the most out of your big data. But remember if you want help, the ideal is to consult an expert.
