Hadoop Commands

Introduction to Apache Hadoop

Apache Hadoop is an open-source framework that allows for the distributed processing of large data sets across clusters of computers using simple programming models. It is designed to scale up from a single server to thousands of machines, each offering local computation and storage. Hadoop is known for its reliability and ability to handle vast amounts of data efficiently.

The Hadoop Distributed File System (HDFS) is a fundamental component of Hadoop, providing scalable and reliable data storage. HDFS is designed to store very large files across a cluster of machines, ensuring fault tolerance and high availability.

Another critical component of Hadoop is the MapReduce programming model, which enables efficient data processing across the distributed storage provided by HDFS. MapReduce breaks down data processing tasks into two main phases: the Map phase, which filters and sorts the data, and the Reduce phase, which performs aggregation and summarization.Together with YARN (Yet Another Resource Negotiator), which manages and schedules resources in the Hadoop cluster, these components make Hadoop a robust platform for big data analytics, allowing organizations to gain insights from their data at unprecedented scales.

Hadoop Installation and Setup Guide

Before starting Hadoop, ensure you have:

• Downloaded Hadoop from the official Hadoop website and Java from the Oracle website. Note that Java is downloaded from Oracle, so if you don’t have an account, you will need to create one.
• Checked if the Hadoop files are already associated with WinRAR (indicated by red box logos on the files). If they are, WinRAR is already installed. If not, you can download WinRAR from the official WinRAR website. After downloading, unzip the Hadoop file into the `C:\Hadoop` directory by right-clicking on the file and selecting "Extract Here."
• Verified if `winutils` is running by opening a Command Prompt and typing winutils.exe. If it is not recognized or does not run, you will need to download the necessary `msvcr120.dll` file. Follow these steps:
  • Go to the msvcr120.dll download page and download the file.
  • Navigate to the `C:\Windows\System32` directory.
  • Copy and paste the downloaded `msvcr120.dll` file into the `System32` folder.
  • After copying the DLL file, try running winutils.exe again in Command Prompt.
• During Java installation, create two folders within the Java directory: one for the JDK (C:\Java\jdk) and one for the JRE (C:\Java\jre). Ensure to update the paths accordingly in the installation wizard.
• Created a folder named Hadoop in the root of the Windows (C:) drive for installing Hadoop.
• Created a `data` folder within the `Hadoop` directory, and inside the `data` folder, created two subfolders: `dfs/namenode` and `dfs/datanode`. These directories are used by Hadoop to store the namenode and datanode data.
• Properly installed Hadoop on the system.
• Configured the necessary environment variables:
  • Set HADOOP_HOME to C:\Hadoop\bin in the system variables.
  • Set JAVA_HOME to the JDK path C:\Java\jdk in system variables.
  • Added C:\Hadoop\bin and C:\Hadoop\sbin to the system PATH variable.
  • Added JAVA_HOME\bin to the user PATH variable.
• Updated the core Hadoop configuration files (core-site.xml, hdfs-site.xml, yarn-site.xml, and mapred-site.xml). Update the bin file in the Hadoop folder. You can download the updated bin folder from here, then copy it directly into your Hadoop directory. For the XML files, navigate to C:\Hadoop\etc\hadoop and download the updated XML files. Just copy and paste them directly into this folder to update.
• Updated the hadoop-env.cmd file to include the JAVA_HOME variable. Set JAVA_HOME to C:\Java\jdk in the C:\Hadoop\etc\hadoop\hadoop-env.cmd file.
• Launch Command Prompt as an Administrator and execute the command hdfs namenode -format. This step is only necessary during the initial setup, as it formats all the namenodes.

Troubleshooting

If your DataNode is not running, follow these steps:

1. DataNode and NameNode Version Mismatch

1. Navigate to C:\Hadoop\data\dfs\current\namenode and open the VERSION file.
2. Copy the ClusterID value from the VERSION file.
3. Go to C:\Hadoop\data\dfs\current\datanode and paste the same ClusterID value into the corresponding VERSION file in this directory.

2. Permission Issues

1. Check if the DataNode directory is not creating the necessary files.
2. Navigate to C:\Hadoop\data\dfs\current\datanode and right-click on the directory.
3. Select Properties, then go to the Security tab.
4. Click on Edit and ensure that the user has full control by selecting Allow for all permissions.
5. Click Apply and then OK to save the changes.

In most cases, these steps will resolve the issue with the DataNode not running.

Example Workflow

1. Start Command Prompt in Admin Mode:
   • Press Windows Key + X, select Command Prompt (Admin), and confirm any prompts.
2. Navigate to the sbin Directory:
   • cd C:\Hadoop\sbin
3. Start HDFS:
   • start-dfs.cmd
4. Start YARN:
   • start-yarn.cmd
5. Verify HDFS:
   • Open http://localhost:9864/ in the web browser for datanode information.
   • Open http://localhost:9870/ in the web browser for namenode information.
6. Verify YARN:
   • Open http://localhost:8088/ in the web browser.

Following these steps will help ensure that Hadoop services are correctly started and running on the local machine. For more detailed information, refer to the official Apache Hadoop documentation. Detailed steps for starting this services are provided later in this page.