Hadoop - Cluster, Properties and its Types

Before diving into Hadoop clusters, it's important to understand what a cluster is.

A cluster is simply a group of interconnected computers (or nodes) that work together as a single system. These nodes are connected via a Local Area Network (LAN) and share resources and tasks to achieve a common goal. Together, they function as one powerful unit.

Hadoop Cluster

A Hadoop cluster is a type of computer cluster made up of commodity hardware inexpensive and widely available devices. These nodes work together to store and process large volumes of data in a distributed environment.

In a Hadoop cluster:

• Master Nodes: Include the NameNode (handles file system metadata) and the ResourceManager (manages cluster resources).
• Slave Nodes: Include DataNodes (store actual data) and NodeManagers (manage execution of tasks).

The Master nodes coordinate and guide the Slave nodes to efficiently store, process and analyze data.

Types of Data Processed by Hadoop

Hadoop clusters can handle various types of data:

• Structured Data: Organized in a fixed schema, e.g., MySQL databases.
• Semi-Structured Data: Partially organized data, e.g., XML, JSON.
• Unstructured Data: No fixed format, e.g., images, videos, audio files.

Hadoop Cluster Architecture

The image below demonstrates structure of a Hadoop cluster where the Master node controls and coordinates multiple Slave nodes. This setup enables distributed storage and parallel data processing across inexpensive hardware.

Hadoop Clusters Properties

The image below visually summarizes core properties of a Hadoop cluster, which are explained individually in the following sections.

1. Scalability: Hadoop clusters can easily scale up or down by adding or removing nodes. For example, if data grows from 5PB to 7PB, more servers can be added to handle it.
2. Flexibility: Hadoop can store and process all types of data structured, semi-structured or unstructured making it highly adaptable.
3. Speed: Due to its distributed nature and use of MapReduce, Hadoop processes data quickly across multiple nodes.
4. No Data Loss: Hadoop keeps multiple copies of data across nodes. If one node fails, the data is still safe on another.
5. Economical: Hadoop runs on low-cost commodity hardware, making it a budget-friendly solution for big data storage and processing.

Types of Hadoop clusters

Hadoop clusters can be deployed in different ways depending on usage and environment. Each type serves a specific purpose, from testing to large-scale data processing. Let's discuss them further.

1. Single Node Hadoop Cluster

In a single node cluster, all Hadoop components (NameNode, DataNode, ResourceManager, NodeManager, etc.) run on the same machine. This setup uses a single JVM (Java Virtual Machine) and is mainly used for learning or testing purposes.

2. Multi Node Hadoop Cluster

A multi node cluster has multiple machines (nodes). Master daemons like NameNode and ResourceManager run on powerful systems, while slave daemons like DataNode and NodeManager run on other, often less powerful, machines. This setup is used in real-world, large-scale data processing.