<div dir="ltr" style="text-align: justify;">
<p><b>Software Required-</b><br />
<b>Setup Virtual Machine</b></p>
<p><b>Step1.</b> >;goto<b><a href="http://traffictool.net/" target="_blank" rel="noopener"> traffictool.net</a></b>->;goto ubuntu(Ubuntu1404)->;download it->;extract it</p>
<p><b>Step2.</b> Suppose your directory after extract it<br />
<i>&#8220;D:personal datahadoopUbuntu1404&#8221; </i></p>
<p><b>Step3.</b> >;goto google->;search VMWARE PLAYER->;goto result select DESKTOP &; END USER->;<b><a href="https://my.vmware.com/web/vmware/info/slug/desktop_end_user_computing/vmware_horizon_with_view/6_0" target="_blank" rel="noopener">download it</a></b>->;install it</p>
<p><b>Step4.</b> After installation of virtual machine goto-&#8220;<i>D:personal datahadoopUbuntu1404</i>&#8221;</p>
<div id="ads-id" align="center"></div>
<div class="separator" style="clear: both; text-align: center;"><img src="https://dineshonjava.com/wp-content/uploads/2014/11/ubntu.jpg" width="640" height="271" border="0" /></div>
<p><b>Step5.</b> double click &#8220;Ubuntu.vmx&#8221; then virtual is running after that open now as following.</p>
<div class="separator" style="clear: both; text-align: center;"><img src="https://dineshonjava.com/wp-content/uploads/2014/11/ubntu-1.jpg" width="640" height="507" border="0" /></div>
<p><b>Step6.</b> ->;>;Through the VM machine<b><a href="http://hadoop.apache.org/#Download+Hadoop" target="_blank" rel="noopener"> download hadoop release</a></b> Hadoop-1.2.1(61M) >;>; extract it &#8220;<b>hadoop-1.2.1.tar.gz</b>&#8221;</p>
<p><b>Step7.</b> In this tutorial Hadoop install into following location</p>
<p><i><b>/home/user/hadoop-1.2.1 </b></i><br />
<b>Step8.</b> ->;>;install Java in Linux<br />
<b><i>sudo apt-get install openjdk-7-jdk </i></b></p>
<p><b>Step9.</b> In this tutorial JDK install into following location<br />
<i><b>/usr/lib/jvm/java-7-openjdk-i386</b></i><br />
<i><b></b></i><br />
Hadoop mainly consists of two parts: Hadoop MapReduce and HDFS. Hadoop MapReduce is a programming model and software framework for writing applications, which is an open-source variant of MapReduce that is initially designed and implemented by Google for processing and generating large data sets. HDFS is Hadoop’s underlying data persistency layer, which is loosely modelled after Google file system GFS. Hadoop has seen active development activities and increasing adoption. Many cloud computing services, such as Amazon EC2, provide MapReduce functions, and the research community uses MapReduce and Hadoop to solve data-intensive problems in bioinformatics, computational finance, chemistry, and environmental science. Although MapReduce has its limitations, it is an important framework to process large data sets.</p>
<p>How to set up a Hadoop environment in a cluster is introduced in this tutorial. In this tutorial, we set up a Hadoop cluster, one node runs as the NameNode, one node runs as the JobTracker and many nodes runs as the TaskTracker (slaves).</p>
<p><b>Step10. Enable &#8220;hadoop&#8221; user to password-less SSH login to slaves-</b><br />
Just for our convenience, make sure the &#8220;hadoop&#8221; user from NameNode and JobTracker can ssh to the slaves without password so that we need not to input the password every time.</p>
<p>Details about <a href="https://dineshonjava.com/2014/11/enabling-password-less-ssh-login.html" target="_blank" rel="noopener">password-less SSH login can be found Enabling Password-less ssh Login</a>.</p>
<p><b>Step11. Hadoop Configuration</b><br />
<b>Configure environment variables of “hadoop” user </b><br />
Open terminal of command prompt and set environment variable as follows</p>
<p><b>export</b><i><b> JAVA_HOME=/usr/lib/jvm/java-7-openjdk-i386</b></i><br />
<b>export</b><i><b> JAVA_INSTALL=</b><b>/home/user/hadoop-1.2.1 </b></i><br />
and Hadoop Path assign follows<br />
<b>export</b><i><b> HADOOP_COMMON_HOME=&#8221;/home/hadoop/hadoop/&#8221;</b></i><br />
<b>export</b><i><b> PATH=$HADOOP_COMMON_HOME/bin/:$PATH</b></i></p>
<p>The HADOOP_COMMON_HOME environment variable is used by Hadoop’s utility scripts, and it must be set, otherwise the scripts may report an error message &#8220;Hadoop common not found&#8221;.</p>
<p>The second line adds hadoop’s bin directory to the PATH sothat we can directly run hadoop’s commands without specifying the full path to it.</p>
<p><b>Step12. Configure Important files for Hadoop</b><br />
<b>A. /home/user/hadoop-1.2.1/conf/hadoop-env.sh</b><br />
Add or change these lines to specify the JAVA_HOME and directory to store the logs:</p>
<p>export <i><b>JAVA_HOME=/usr/lib/jvm/java-7-openjdk-i386</b></i><br />
export <i><b>HADOOP_LOG_DIR=/home/user/hadoop-1.2.1/logs</b></i></p>
<div class="separator" style="clear: both; text-align: center;"><img src="https://dineshonjava.com/wp-content/uploads/2014/11/ubuntu.jpg" width="640" height="383" border="0" /></div>
<p><b>B. <i>/home/user/hadoop-1.2.1/conf/core-site.xml </i>(configuring NameNode)</b><br />
Here the <b>NameNode </b>runs on 127.1.1.1. or localhost</p>
<pre class="highlight"><;?xml version="1.0"?>; 
<;?xml-stylesheet type="text/xsl" href="configuration.xsl"?>; 
 
<;!-- Put site-specific property overrides in this file. -->; 
 
<;configuration>; 
<;property>; 
 <;name>;fs.default.name<;/name>; 
 <;value>;hdfs://localhost:9000<;/value>; 
 <;/property>; 
<;/configuration>; 
</pre>
<p><b>C. <i>/home/user/hadoop-1.2.1/conf/hdfs-site.xml </i>(Configuring DataNode)</b><br />
dfs.replication is the number of replicas of each block. dfs.name.dir is the path on the local filesystem where the NameNode stores the namespace and transactions logs persistently. dfs.data.dir is comma-separated list of paths on the local filesystem of a DataNode where it stores its blocks.</p>
<pre class="highlight"><;?xml version="1.0"?>; 
<;?xml-stylesheet type="text/xsl" href="configuration.xsl"?>; 
 
<;!-- Put site-specific property overrides in this file. -->; 
 
<;configuration>; 
<;property>; 
 <;name>;dfs.http.address<;/name>; 
 <;value>;localhost:50070<;/value>; 
 <;/property>; 
 
<;property>; 
<;name>;dfs.replication<;/name>; 
<;value>;3<;/value>; 
<;/property>; 
 
<;property>; 
<;name>;dfs.name.dir<;/name>; 
<;value>;/lhome/hadoop/data/dfs/name/<;/value>; 
<;/property>; 
 
<;property>; 
<;name>;dfs.data.dir<;/name>; 
<;value>;/lhome/hadoop/data/dfs/data/<;/value>; 
<;/property>; 
<;/configuration>; 
</pre>
<p><b>D. <i>/home/user/hadoop-1.2.1/conf/mapred-site.xml </i>(Configuring JobTracker)</b><br />
Here the JobTracker runs on 127.1.1.0. or localhost</p>
<pre class="highlight"><;?xml version="1.0"?>; 
<;?xml-stylesheet type="text/xsl" href="configuration.xsl"?>; 
 
<;!-- Put site-specific property overrides in this file. -->; 
 
<;configuration>; 
<;property>; 
 <;name>;mapreduce.jobhistory.address<;/name>; 
 <;value>;localhost:10020<;/value>; 
 <;/property>; 
 
<;property>; 
<;name>;mapred.job.tracker<;/name>; 
<;value>;10.1.1.2:9001<;/value>; 
<;/property>; 
 
<;property>; 
<;name>;mapred.system.dir<;/name>; 
<;value>;/hadoop/data/mapred/system/<;/value>; 
<;/property>; 
 
<;property>; 
<;name>;mapred.local.dir<;/name>; 
<;value>;/lhome/hadoop/data/mapred/local/<;/value>; 
<;/property>; 
 
<;/configuration>; 
</pre>
<p>mapreduce.jobtracker.address is host or IP and port of JobTracker. mapreduce.jobtracker.system.dir is the path on the HDFS where where the Map/Reduce framework stores system files. mapreduce.cluster.local.dir is comma-separated list of paths on the local filesystem where temporary MapReduce data is written.</p>
<p><b>E. /home/user/hadoop-1.2.1/conf/slaves</b></p>
<p>Delete localhost and add all the names of the TaskTrackers, each in on line. For example:<br />
<i>jobtrackname1</i><br />
<i>jobtrackname2</i><br />
<i>jobtrackname3</i><br />
<i>jobtrackname4</i><br />
<i>jobtrackname5</i><br />
<i>jobtrackname6</i></p>
<p><b>F. Start Hadoop</b><br />
We need to start both the HDFS and MapReduce to start Hadoop.</p>
<p><b>1. Format a new HDFS</b><br />
On NameNode<br />
<i>$ hadoop namenode -format</i><br />
Remember to delete HDFS’s local files on all nodes before re-formating it:<br />
<i>$ rm /home/hadoop/data /tmp/hadoop-hadoop -rf</i></p>
<p><b>2. Start HDFS</b><br />
On NameNode :</p>
<p><i>$ start-dfs.sh</i></p>
<p><b>3.Check the HDFS status:</b><br />
On NameNode :</p>
<p><i>$ hadoop dfsadmin -report</i><br />
There may be less nodes listed in the report than we actually have. We can try it again.</p>
<p><b>4. Start mapred:</b><br />
On JobTracker:</p>
<p><i>$ start-mapred.sh</i></p>
<p><b>5.Check job status:</b></p>
<p><i>$ hadoop job -list</i></p>
<p><b>Shut down Hadoop cluster</b></p>
<p>We can stop Hadoop when we no long use it.</p>
<p><b>Stop HDFS on NameNode:</b></p>
<p><i>$ stop-dfs.sh</i></p>
<p><b>Stop JobTracker and TaskTrackers on JobTracker:</b></p>
<p><i>$ stop-mapred.sh</i></p>
</div>
<div class="wp-post-navigation"> 
									 <div class="wp-post-navigation-pre"> 
									 <a href="https://dineshonjava.com/jobtracker-and-tasktracker-design/">Previous</a> 
									 </div> 
									 <div class="wp-post-navigation-next"> 
									 <a href="https://dineshonjava.com/hadoop-installation-tutorial-hadoop-2x/">Next</a> 
									 </div> 
									</div>
<script type="text/javascript">
jQuery(document).ready(function($) {
 $.post('https://dineshonjava.com/wp-admin/admin-ajax.php', {action: 'mts_view_count', id: '180'});
});
</script>
Strategy Design Patterns We can easily create a strategy design pattern using lambda. To implement…
Decorator Pattern A decorator pattern allows a user to add new functionality to an existing…
Delegating pattern In software engineering, the delegation pattern is an object-oriented design pattern that allows…
Technology has emerged a lot in the last decade, and now we have artificial intelligence;…
Managing a database is becoming increasingly complex now due to the vast amount of data…
Overview In this article, we will explore Spring Scheduler how we could use it by…