How do I set up Hadoop Cluster for learning Hadoop using Laptop?

Setting up a Hadoop cluster on a laptop for learning purposes involves creating a small-scale, single-node cluster that mimics the distributed nature of Hadoop. Here are simplified steps to set up a Hadoop cluster on your laptop:

Prerequisites: Before you begin, ensure that you have the following prerequisites installed on your laptop:

1. Java Development Kit (JDK):

   • Install Java, as Hadoop is a Java-based framework. You can download and install the JDK from the official Oracle website or use OpenJDK.

2. SSH:

   • Hadoop relies on SSH for communication between nodes. Ensure that SSH is installed and configured on your laptop.

3. Hadoop Binary:

   • Download the Hadoop binary distribution from the official Apache Hadoop website (https://hadoop.apache.org/). Choose a stable version and download the tarball.

Steps to Set Up a Single-Node Hadoop Cluster:

1. Extract Hadoop Tarball:

   • Extract the downloaded Hadoop tarball to a directory of your choice. This will be your Hadoop home directory.
   bash

• tar -zxvf hadoop-x.x.x.tar.gz

• Configure Hadoop:

  • Navigate to the Hadoop home directory and configure Hadoop by editing the configuration files. Key configuration files include core-site.xml, hdfs-site.xml, and mapred-site.xml in the etc/hadoop/ directory.

  • Configure core-site.xml:

    xml
• <configuration> <property> <name>fs.defaultFS</name> <value>hdfs://localhost:9000</value> </property> </configuration>

• Configure hdfs-site.xml:

  xml
• <configuration> <property> <name>dfs.replication</name> <value>1</value> </property> </configuration>

• Rename mapred-site.xml.template to mapred-site.xml and configure it:

  xml
• • <configuration> <property> <name>mapreduce.framework.name</name> <value>yarn</value> </property> </configuration>

• Format HDFS:

  • Format the Hadoop Distributed File System (HDFS) by running the following command:
  bash
• bin/hdfs namenode -format

• Start Hadoop Services:

  • Start the Hadoop services. Run the following commands:
  bash
• sbin/start-dfs.sh sbin/start-yarn.sh

• Verify Hadoop Installation:

  • Open a web browser and go to the following URLs to verify that Hadoop services are running:
    • HDFS NameNode: http://localhost:9870/
    • YARN ResourceManager: http://localhost:8088/

• Run a Sample MapReduce Job:

  • Run a sample MapReduce job to ensure that the Hadoop cluster is functioning correctly:
  bash
• bin/hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-x.x.x.jar pi 10 100

  This command estimates the value of pi using a MapReduce job.

• Stop Hadoop Services:

  • Stop the Hadoop services when you're done:
  bash