<div dir="ltr" style="text-align: justify;" trbidi="on">
<div dir="ltr" style="text-align: justify;" trbidi="on">In the Hadoop and MapReduce tutorial we will see how to create hello world job and what are the steps to creating a mapreduce program. There are following steps to creating mapreduce program.</p>
<p><b>Step1- Creating a file</b><br />
<b>J$ <i>cat>;file.txt</i></b><br />
<i>hi how are you</i><br />
<i>how is your job</i><br />
<i>how is your family</i><br />
<i>how is your brother </i><br />
<i>how is your sister</i><br />
<i>what is the time now </i><br />
<i>what is the strength of hadoop</i></p>
<div align='center' id="ads-id"></div>
<p><b>Step2- loading file.txt from local file system to HDFS</b><br />
<b>J$</b><i><b> hadoop fs -put file.txt ; file</b></i></p>
<p><b>Step3- Writing programs</b></p>
<ol style="text-align: left;">
<li> ;<i>DriverCode.java</i></li>
<li><i> ;MapperCode.java</i></li>
<li><i> ;ReducerCode.java</i></li>
</ol>
<p>
<b>Step4- Compiling all above .java files</b></p>
<p>J$<i> javac -classpath $HADOOP_HOME/hadoop-core.jar *.java</i></p>
<p><b>Step5- Creating jar file</b></p>
<p><b>J$<i> jar cvf job.jar *.class</i></b></p>
<p><b>Step6- Running above job.jar on file (which there in HDFS)</b></p>
<p>J$<i> hadoop jar job.jar DriverCode file TestOutput</i></p>
<p>Lets start with actual code for these steps above.</p>
<p>Hello World Job ->; WordCountJob</p>
<p><b>1. DriverCode (WordCount.java)</b></div>
<pre class="highlight" name="code">package com.doj.hadoop.driver;

import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapred.FileInputFormat;
import org.apache.hadoop.mapred.FileOutputFormat;
import org.apache.hadoop.mapred.JobClient;
import org.apache.hadoop.mapred.JobConf;
import org.apache.hadoop.mapred.Mapper;
import org.apache.hadoop.mapred.Reducer;
import org.apache.hadoop.mapred.TextInputFormat;
import org.apache.hadoop.mapred.TextOutputFormat;

/**
 * @author Dinesh Rajput
 *
 */
public class WordCount extends Configured implements Tool {
 @Override
 public int run(String[] args) throws Exception {
 if (args.length != 2) {
 System.err.printf("Usage: %s [generic options] <;input>; <;output>;n",
 getClass().getSimpleName());
 ToolRunner.printGenericCommandUsage(System.err);
 return -1;
 }
 JobConf conf = new JobConf(WordCount.class);
 conf.setJobName("Word Count");
 FileInputFormat.addInputPath(conf, new Path(args[0]));
 FileOutputFormat.setOutputPath(conf, new Path(args[1]));
 conf.setMapperClass(WordMapper.class);
 conf.setCombinerClass(WordReducer.class);
 conf.setReducerClass(WordReducer.class);
 conf.setOutputKeyClass(Text.class);
 conf.setOutputValueClass(IntWritable.class);
 return conf.waitForCompletion(true) ? 0 : 1;
 }
 public static void main(String[] args) throws Exception {
 int exitCode = ToolRunner.run(new WordCount(), args);
 System.exit(exitCode);
 }
}
</pre>
<p><b>2. MapperCode (WordMapper.java)</b></p>
<pre class="highlight" name="code">package com.doj.hadoop.driver;

import java.io.IOException;
import java.util.StringTokenizer;
 
import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.LongWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapred.MapReduceBase;
import org.apache.hadoop.mapred.Mapper;
import org.apache.hadoop.mapred.OutputCollector;
import org.apache.hadoop.mapred.Reporter;

/**
 * @author Dinesh Rajput
 *
 */
public class WordMapper extends MapReduceBase implements Mapper<;LongWritable, Text, Text, IntWritable>;
{
 private final static IntWritable one = new IntWritable(1);
 
 private Text word = new Text();
 
 public void map(LongWritable key, Text value, OutputCollector<;Text, IntWritable>; output, Reporter reporter)
 throws IOException
 {
 String line = value.toString();
 StringTokenizer tokenizer = new StringTokenizer(line);
 while (tokenizer.hasMoreTokens())
 {
 word.set(tokenizer.nextToken());
 output.collect(word, one);
 }
 }
}

</pre>
<p>
<b>3. ReducedCode (WordReducer.java)</b></p>
<pre class="highlight" name="code">package com.doj.hadoop.driver;

/**
 * @author Dinesh Rajput
 *
 */
import java.io.IOException;
import java.util.Iterator;
 
import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapred.MapReduceBase;
import org.apache.hadoop.mapred.OutputCollector;
import org.apache.hadoop.mapred.Reducer;
import org.apache.hadoop.mapred.Reporter;
 
public class WordReducer extends MapReduceBase implements Reducer<;Text, IntWritable, Text, IntWritable>;
{
 public void reduce(Text key, Iterator<;IntWritable>; values, OutputCollector<;Text, IntWritable>; output,
 Reporter reporter) throws IOException
 {
 int sum = 0;
 while (values.hasNext())
 {
 sum += values.next().get();
 }
 output.collect(key, new IntWritable(sum));
 }
}

</pre>
</div>
<div class="wp-post-navigation"> 
									 <div class="wp-post-navigation-pre"> 
									 <a href="https://dineshonjava.com/mapreduce-flow-chart-sample-example/">Previous</a> 
									 </div> 
									 <div class="wp-post-navigation-next"> 
									 <a href="https://dineshonjava.com/hadoop-tutorial/">Next</a> 
									 </div> 
									</div>
<script type="text/javascript">
jQuery(document).ready(function($) {
 $.post('https://dineshonjava.com/wp-admin/admin-ajax.php', {action: 'mts_view_count', id: '174'});
});
</script>
Strategy Design Patterns We can easily create a strategy design pattern using lambda. To implement…
Decorator Pattern A decorator pattern allows a user to add new functionality to an existing…
Delegating pattern In software engineering, the delegation pattern is an object-oriented design pattern that allows…
Technology has emerged a lot in the last decade, and now we have artificial intelligence;…
Managing a database is becoming increasingly complex now due to the vast amount of data…
Overview In this article, we will explore Spring Scheduler how we could use it by…