<div dir="ltr" style="text-align: justify;">
<p>In this mapreduce tutorial we will explain mapreduce sample example with its flow chart. How to work mapreduce for a job.<br />
<b></b></p>
<h2><b>A SIMPLE EXAMPLE FOR WORD COUNT</b></h2>
<ul style="text-align: left;">
<li style="list-style-type: none;">
<ul style="text-align: left;">
<li>We have a large collection of text documents in a folder. (Just to give a feel size.. we have 1000 documents each with average of 1 Millions words)</li>
<li><i><b>What we need to calculate:-</b></i>
<ul>
<li><i><b>Count the frequency of each distinct word in the documents?</b></i></li>
</ul>
</li>
</ul>
</li>
</ul>
<p> ;</p>
<ul style="text-align: left;">
<li style="list-style-type: none;">
<ul style="text-align: left;">
<li><i><b>How would you solve this using simple Java program?</b></i></li>
</ul>
</li>
</ul>
<p> ;</p>
<ul style="text-align: left;">
<li style="list-style-type: none;">
<ul style="text-align: left;">
<li><i><b>How many lines of codes will u write?</b></i></li>
</ul>
</li>
</ul>
<p> ;</p>
<ul style="text-align: left;">
<li><i><b>How much will be the program execution time?</b></i></li>
</ul>
<p>To overcome listed above problems into some line using mapreduce program. Now we look into below mapreduce function for understanding how to its work on large dataset.</p>
<h2><b>MAP FUNCTION</b></h2>
<ul style="text-align: left;">
<li>Map Functions operate on every key, value pair of data and transformation logic provided in the map function.</li>
<li>Map Function always emits a Key, Value Pair as output<br />
<b>Map(Key1, Valiue1) &#8211;>; List(Key2, Value2)</b></li>
<li>Map Function transformation is similar to Row Level Function in Standard SQL</li>
<li>For Each File
<ul>
<li><b>Map Function is</b>
<ul>
<li>Read each line from the input file
<ul>
<li>Tokenize and get each word
<ul>
<li>Emit the word, 1 for every word found</li>
</ul>
</li>
</ul>
</li>
</ul>
</li>
</ul>
</li>
</ul>
<div id="ads-id" align="center"></div>
<p><b>The emitted word, 1 will from the List that is output from the mapper</b></p>
<p><b>So who take ensuring the file is distributed and each line of the file is passed to each of the map function?</b>-Hadoop Framework take care about this, no need to worry about the distributed system.</p>
<h2><b>REDUCE FUNCTION</b></h2>
<ul style="text-align: left;">
<li>Reduce Functions takes list of value for every key and transforms the data based on the (aggregation) logic provided in the reduce function.</li>
<li>Reduce Function<br />
<b>Reduce(Key2, List(Value2)) &#8211;>; List(Key3, Value3)</b></li>
<li>Reduce Functions is similar to Aggregate Functions in Standard SQL</li>
</ul>
<p><b>Reduce(Key2, List(Value2)) &#8211;>; List(Key3, Value3)</b></p>
<p>For the List(key, value) output from the mapper Shuffle and Sort the data by key<br />
Group by Key and create the list of values for a key</p>
<ul style="text-align: left;">
<li><b>Reduce Function is</b>
<ul>
<li>Read each key (word) and list of values (1,1,1..) associated with it.
<ul>
<li>For each key add the list of values to calculate sum
<ul>
<li>Emit the word, sum for every word found</li>
</ul>
</li>
</ul>
</li>
</ul>
</li>
</ul>
<p><b>So who is ensuring the shuffle, sort, group by etc?</b></p>
<h2><b>MAP FUNCTION FOR WORD COUNT</b></h2>
<pre class="highlight">private final static IntWritable one = new IntWritable(1); 
private Text word = new Text(); 
 
public void map(LongWritable key, Text value, Context context) throws IOException, InterruptedException { 
String line = value.toString(); 
StringTokenizer tokenizer = new StringTokenizer(line); 
While(tokenizer.hasMoreTokens()){ 
 
word.set(tokenizer.nextToken()); 
context.write(word, one); 
} 
} 
</pre>
<h2><b>REDUCE FUNCTION FOR WORD COUNT</b></h2>
<pre class="highlight">public void reduce(Text key, Iterable <;IntWritable>; values, Context context) throws IOException, InterruptedException{ 
 
int sum = 0; 
for(IntWritable val : values){ 
sum += val.get(); 
} 
context.write(key, new IntWritable(sum)); 
} 
</pre>
<h2><b>ANATOMY OF A MAPREDUCE PROGRAM</b></h2>
<div class="separator" style="clear: both; text-align: center;"><img src="https://dineshonjava.com/wp-content/uploads/2014/11/map-reduce.jpg" border="0" /></div>
<h2><b>FLOW CHART OF A MAPREDUCE PROGRAM</b></h2>
<p>Suppose we have a file with size about 200 MB, suppose content as follows</p>
<p><b>&#8212;&#8212;&#8212;&#8211;file.txt&#8212;&#8212;&#8212;&#8212;</b><br />
<b>_______File(200 MB)____________</b><br />
<i>hi how are you</i><br />
<i>how is your job (64 MB) 1-Split</i><br />
<i>________________________________</i><br />
<i>&#8212;&#8212;&#8212;&#8212;&#8212;&#8212;&#8212;&#8212;&#8212;&#8212;-</i><br />
<i>________________________________</i><br />
<i>how is your family</i><br />
<i>how is your brother (64 MB) 2-Split</i><br />
<i>________________________________</i><br />
<i>&#8212;&#8212;&#8212;&#8212;&#8212;&#8212;&#8212;&#8212;&#8212;&#8212;-</i><br />
<i>________________________________</i><br />
<i>how is your sister</i><br />
<i>what is the time now (64 MB) 3-Split</i><br />
<i>________________________________</i><br />
<i>&#8212;&#8212;&#8212;&#8212;&#8212;&#8212;&#8212;&#8212;&#8212;&#8212;-</i><br />
<i>_______________________________</i><br />
<i>what is the strength of hadoop (8 MB) 4-Split</i><br />
<i>________________________________</i><br />
<i>&#8212;&#8212;&#8212;&#8212;&#8212;&#8212;&#8212;&#8212;&#8212;&#8212;-</i></p>
<p>In above file we have divided this file into 4 splits with sizes three splits with size 64 MB and last fourth split with size 8 MB.</p>
<p><b>Input File Formats:</b><br />
&#8212;&#8212;&#8212;&#8212;&#8212;&#8212;&#8212;&#8212;&#8212;-<br />
<b>1. TextInputFormat</b><br />
<b>2. KeyValueTextInputFormat</b><br />
<b>3. SequenceFileInputFormat</b><br />
<b>4. SequenceFileAsTextInputFormat </b><br />
&#8212;&#8212;&#8212;&#8212;&#8212;&#8212;&#8212;&#8212;&#8212;&#8212;</p>
<div class="separator" style="clear: both; text-align: center;"><img src="https://dineshonjava.com/wp-content/uploads/2014/11/mapreduce-flow-chart.jpg" width="640" height="538" border="0" /></div>
<p><b>Lets see in another following figure to understand the process of MAPREDUCE.</b></p>
<div class="separator" style="clear: both; text-align: center;"><img src="https://dineshonjava.com/wp-content/uploads/2014/11/mapreduce-process.jpg" width="640" height="289" border="0" /></div>
</div>
<div class="wp-post-navigation"> 
									 <div class="wp-post-navigation-pre"> 
									 <a href="https://dineshonjava.com/introduction-to-mapreduce/">Previous</a> 
									 </div> 
									 <div class="wp-post-navigation-next"> 
									 <a href="https://dineshonjava.com/mapreduce-programming-hello-world-job/">Next</a> 
									 </div> 
									</div>
<script type="text/javascript">
jQuery(document).ready(function($) {
 $.post('https://dineshonjava.com/wp-admin/admin-ajax.php', {action: 'mts_view_count', id: '175'});
});
</script>
Strategy Design Patterns We can easily create a strategy design pattern using lambda. To implement…
Decorator Pattern A decorator pattern allows a user to add new functionality to an existing…
Delegating pattern In software engineering, the delegation pattern is an object-oriented design pattern that allows…
Technology has emerged a lot in the last decade, and now we have artificial intelligence;…
Managing a database is becoming increasingly complex now due to the vast amount of data…
Overview In this article, we will explore Spring Scheduler how we could use it by…