Testing Hadoop Mapreduce Programs With MRUnit

Our HadoopTpoint App is now available in google play store,please rate and comment it in play store : W3Schools

Generally Distributed Programmings like Mapreduce programs are very difficult to debug after completing the entire code,so that kind of programs better to test in earlier stages.This testing is very useful to find the bugs and fix the bugs in begging stages of programming.So Here we are discussing about  Testing Hadoop Mapreduce Programs With MRUnit  .

Generally we are testing the java programs with the help of eclipse and Junit plugin and now we can test the mapreduce jobs with the help of MRUnit.In this testing processes we don’t need to use HDFS and don’t need to start Mapreduce related demons.

Here i am using Eclipse Luna and Hadoop 2.6.0 version so i added 3 extra jar files to eclipse project folder for Testing Hadoop Mapreduce Programs With MRUnit.(Before you should added required hadoop jar files to your eclipse) Here is the required jar files for MRunit testing.

preparing MRUnit Test steps

  • Initialize Mapper,reducer and Mapreduce classes
  • setup()
  • Test Mapper
  • Test Reducer
  • Test Mapreducer

Preparing  Test

We will initialize the Mapper and Reducer classes. MapReduceDriver specification list out the Map input key – value, Map output key – value, Reduce output key – value types. In thesetup(),  set the Mapper and Reducer classes on the MapReduceDriver.

 

Testing Hadoop Mapreduce Programs With MRUnit

Testing Mapper

Pass the input key and input value of the Mapper to   mapDriver.withInput  and pass the expected output key and output value from the Mapper to  mapDriver.withOutput  for the given input key and input value.

mapDriver.runTest()  will run the configured Mapper with the input and the expected output. If the output from the Mapper for the given input is same as the provided output the test will report a success or it will fail.

Testing Reducer

Similar to the Mapper test, we will specify the input and the expected output for the reducer using reduceDriver.withInput  and  reduceDriver.withOutput . If the output from the Reducer for the given input is same as the provided output the test will report a success or it will fail. reduceDriver.runTest() is used for test the reducer driver.

Testing Mapreduce

In this step we specify the input and expected output by using  mapReduceDriver.withInput and mapReduceDriver.withOutput . if the output from the mapreduce for the given input is same as the provided output the test will report a success or it will fail.

 

Original Mapreduce Program

This is the example for Testing Hadoop Mapreduce Programs With MRUnit

Speak Your Mind

*