Job Initialisation in MapReduce

Before a MapReduce job can start processing data, it goes through a crucial initialization phase. This phase prepares the system by setting up resources, validating input/output paths and coordinating with key Hadoop components like the ResourceManager and ApplicationMaster.

This setup ensures that job runs smoothly across the distributed cluster. Let’s explore what happens step-by-step when a MapReduce job is launched.

Step 1: Submitting the Job

When a client submits a MapReduce job, it calls the submitApplication() method.

• This call goes to the Resource Manager (RM).
• The RM hands over the request to the YARN scheduler.
• The scheduler allocates a container on a node (under the Node Manager) where the Application Master (AM) will run.

Step 2: Starting Application Master

The Application Master (AM) used in MapReduce is called MRAppMaster. This master controls the whole job, it keeps track of progress and assigns tasks to different nodes.

During startup, the AM:

• Sets up tools to monitor the job’s progress
• Gets ready to receive updates from tasks as they run

Step 3: Input Splits - Breaking Up Data

Next, the job needs to know how to divide the data:

• Input splits are calculated on the client using data from the shared filesystem (e.g., HDFS).
• For each input split, a Map task object is created.

Example:

If your file is 256MB and the HDFS block size is 128MB --> 2 input splits --> 2 Map tasks

Step 4: Creating Reduce Tasks

The number of Reducer tasks is set using property:

mapreduce.job.reduces

You can configure it in code with:

job.setNumReduceTasks(n)

Choosing Right Reducer Count:

• If not explicitly set, Hadoop may default to 1 reducer which can slow down large jobs.
• Each reducer gets its own task object.
• These reducers process grouped key-value pairs after Shuffling phase.
• Choosing the right number improves parallelism, load balancing and aggregation efficiency.

This diagram shows the key steps - submitting the job, setting reducer count and launching Application Master.

Step 5: Task Assignment Strategy

After all Map and Reduce tasks are created, each is assigned a unique Task ID. The Application Master (AM) decides where to run each task:

• By default: Tasks run in separate containers across the cluster.
• For small jobs: Tasks can run in the same JVM as the AM called an Uber Task, which saves time and resources.

Note: To improve performance, AM schedules Map tasks on the same nodes as the data. This is called data locality and reduces network use.

Uber Task

An Uberised job is one where all tasks run in the same JVM as the AM saving overhead for small jobs.

Conditions for Uberization:

• Job has < 10 mappers
• Only 1 reducer
• Input size is smaller than one HDFS block

Uber task behavior can be controlled using following configuration properties:

• mapreduce.job.ubertask.enable: Enable/disable Uber mode
• mapreduce.job.ubertask.maxmaps: Max number of mappers allowed for Uber
• mapreduce.job.ubertask.maxreduces: Max number of reducers allowed
• mapreduce.job.ubertask.maxbytes: Max input size (in bytes)

Step 6: Output Setup

Before tasks begin execution, the Application Master sets up the output path. It calls setupJob() on the OutputCommitter.

This sets up:

• A final output directory where the results will be stored
• A temporary directory for each task to write its intermediate output

Note: Temporary directories prevent partial or corrupted output in case of task failure. Once a task finishes successfully, temp output is safely committed to the final directory.

By default, Hadoop uses FileOutputCommitter, which manages the creation and cleanup of these files.