Hadoop MapReduce一Knowing the Basics

MapReduce can means two different things:

• Programming Model
• If you can rewrite algorithms into Maps and Reduces, and your problem can be broken up into small pieces solvable in parallel, then MapReduce might be a potential distributed problem solving approach to your large datasets.
• See [1] for a sample MapReduce application
• Software Framework
• A framework such as Hadoop MapRedue breaks up large data into smaller parallelizable chunks and handles scheduling
• Alternative一Apache Tez can process certain workloads more efficiently than MapReduce

In this article, we will learn two kinds of Hadoop MapReduce frameworks provided on Apache Hadoop:

• MapReduce 1 (MR1) 
• YARN (MR2)

We will start the introduction of Hadoop MapReduce using MR1 and then briefly with MR2.

MapReduce Job

A MapReduce job usually splits the input data-set into independent chunks which are processed by the map tasks in a completely parallel manner. The framework sorts the outputs of the maps, which are then input to the reduce tasks. Typically both the input and the output of the job are stored in a file-system (e.g., HDFS on Apache Hadoop). The framework takes care of scheduling tasks, monitoring them and re-executes the failed tasks.

(input)  -> map -> -> combine -> -> reduce ->  (output)

MapReduce job is useful for batch processing on terabytes or petabytes of data stored in Apache Hadoop and has the following characteristics:^[6]

• Cannot control the order in which the maps or reductions are run
• For maximum parallelism, you need Maps and Reduces to not depend on data generated in the same MapReduce job (i.e. stateless) 
• A database with an index will always be faster than a MapReduce job on unindexed data
• Reduce operations do not take place until all Maps are complete (or have failed then been skipped) 
• General assumption that the output of Reduce is smaller than the input to Map一large datasource used to generate smaller final values

Hadoop MapReduce Framework

It's easy to execute MapReduce applications on Apache Hadoop.  Other than simplicity, scalability, and performance, Hadoop MapReduce framework also provides additional benefits such as:

• Failure and recovery
• Minimal data motion

Failure & Recovery

The framework takes care of failures. It is designed to detect and handle failures at the application layer, so delivering a highly-available service on top of a cluster of servers, each of which may be prone to failures.

If a server with one copy of the data is unavailable, another server has a copy of the same key/value pair, which can be used to solve the same sub-task. The JobTracker (see below) keeps track of it all.

Minimal data motion

Typically the compute nodes and the storage nodes are the same, that is, the MapReduce framework and the Hadoop Distributed File System are running on the same set of nodes (see the above diagram). This configuration allows the framework to effectively schedule tasks on the nodes where data is already present, resulting in lower network I/O and very high aggregate bandwidth across the cluster.

Job Tracker & Task Tracker

JobTracker

Applications using MapReduce framework are required to

• Specify Job configuration
• Specify input/output locations
• Supply map, combine and reduce functions

The Hadoop job client then submits the job (jar/executable etc.) and configuration to the JobTracker. In MR1, JobTracker manages and monitors both the resources and the MapReduce Job and task scheduling, which makes the JobTracker a single point of failure (SPOF) in a cluster.  Also, the cluster cannot be scaled efficiently.

TaskTracker 

The JobTracker will first determine the number of splits from the input path and select some TaskTracker based on their network proximity to the data sources, then the JobTracker send the task requests to those selected TaskTrackers.

The TaskTracker spawns a separate JVM processes to do the actual work; this is to ensure that process failure does not take down the task tracker. The TaskTracker monitors these spawned processes, capturing the output and exit codes. When the process finishes, successfully or not, the tracker notifies the JobTracker.

The TaskTrackers also send out heartbeat messages to the JobTracker, usually every few minutes, to reassure the JobTracker that it is still alive. These message also inform the JobTracker of the number of available slots, so the JobTracker can stay up to date with where in the cluster work can be delegated.

YARN一MR2

To overcome the drawbacks of MR1, YARN (or MR2) was introduced and provided for:^[9]

• Better scalability 
• As the JobTracker functionality is split between the ResourceManager and ApplicationMaster.  Hadoop may be run on much larger clusters on MR2.
• Better cluster utilization 
• As the resource capacity configured for each node may be used by both Map and Reduce tasks
• Non-MapReduce clusters 
• May be run on the same cluster concurrently
• Higher throughput 
• Uses finer-grained resource scheduling

References

1. Volume Rendering using MapReduce
2. MapReduce Tutorial (Apache Hadoop)
3. Hadoop MapReduce Framework (Hortonworks)
4. How Hadoop Map/Reduce works
5. Mapper 
• Maps input key/value pairs to a set of intermediate key/value pairs
6. Object-oriented framework presentation (CSCI 5448 Casey McTaggart)
7. YARN Architecture Guide
• The MapReduce framework consists of a single master ResourceManager, one slave NodeManager per cluster-node, and MRAppMaster per application (see tutorial).
• The ResourceManager has two main components: Scheduler and ApplicationsManager.
• Two scheduler plug-ins
• CapacityScheduler and the FairScheduler
8. Mastering Apache Spark 2
9. Practical Hadoop Ecosystem: A Definitive Guide to Hadoop-Related Frameworks
10. MapReduce Tutorial (Apache Hadoop)
• User Interfaces
11. All Cloud-related articles on Xml and More

Apache Ambari一Knowing the Basics

Apache Ambari provides an end-to-end management and monitoring application for Apache Hadoop. In a nutshell, it can be used for:

• Managing most of the administration activities in a Hadoop cluster
• To install, provision, deploy, manage, and monitor a Hadoop cluster
• To hide the complexity of the Hadoop cluster management 
• To provide a very easy and intuitive web UI.
•  Integrating with other external tools for better management via its RESTful APIs

In this article, we will use Apache Ambari (Version 2.2.2.0) from the Hortonworks Data Platform (HDP) in our discussion.

Hortonworks Data Platform

You can deploy Hortonworks Data Platform (HDP) using either Apache Ambari or not.  If you choose not to use Ambari, you can follow the instructions here.  However, it will be much easier to deploy Apache Hadoop stack with Ambari (see the instruction here).

After initial installation and deployment, your Apache Hadoop cluster could still grow and change with use over the time.  With Apache Ambari, you can easily and quickly add new services or expand the storage and processing capacity of the cluster.

The ecosystem of Ambari consists of three main components:

• Ambari Web
• Ambari Server
•  Serves as the collection point for data from across the cluster
• Ambari Agent
• Run on each host in the cluster to allow the Ambari Server to control it

Ambari Web

Using the Ambari Web UI and REST APIs, you can deploy, operate, manage configuration changes, and monitor services for all nodes in your cluster from a central point.

Ambari Web is a client-side JavaScript application, which calls the Ambari REST API (accessible from the Ambari Server) to access cluster information and perform cluster operations. A relational database is used to store the information about the cluster configuration and topology.

With Ambari Views, you can customize the Ambari Web UI.  Ambari Views offer a systematic way to plug-in UI capabilities to surface custom visualization, management and monitoring features in Ambari Web.

Ambari Server

Before starting the Ambari Server, you must set up the Ambari Server once. Setup configures Ambari to talk to the Ambari database, installs the JDK and allows you to customize the user account (default: root) the Ambari Server daemon will run as.

After setup, all the configuration is stored in: 

• /etc/ambari-server/conf/ambari.properties

Then you can run the following commands from the Ambari Server host:

• ambari-server start
• If you reboot your cluster, you must restart the Ambari Server and all the Ambari Agents manually.
• ambari-server status
• ambari-server stop

Once started, you can access Ambari using the following URL:

http://{ambari-server-hostname}:8080

from a web browser.

The start script /usr/sbin/ambari-server is a shell script, that set environment variables and kicks off a python script which kicks off a java process (see details here).

Trobleshooting

You can start ambari in debug mode to get more detailed output via:

ambari-server start --verbose --debug
# or for short
ambari-server start -v -g

Significant files/directories:

• /var/log/ambari-server/ambari-server.log 
• To monitor Ambari, you do
• tail -f /var/log/ambari-server/ambari-server.log
• /var/lib/ambari-server/resources/ 
• SQL scripts to initialize psql DB

Ambari Agent

Ambari Agents will heartbeat to the master every few seconds and will receive commands from the master in the heartbeat responses. Heartbeat responses will be the only way for master to send a command to the Agent. The command will be queued in the action queue, which will be picked up by the action executioner.

Action executioner will pick the right tool (Puppet, Python, etc) for execution depending on the command type and action type. Thus the actions sent in the hearbeat response will be processed asynchronously at the Agent. The action executioner will put the response or progress messages on the message queue. The Agent will send everything on the message queue to the master in the next heartbeat.

Here are the steps you do to install Ambari Agent manually on RHEL/CentOS/Oracle Linux 6:

1. Install the Ambari Agent on every host in your cluster.
• yum install ambari-agent
2. Using a text editor, configure the Ambari Agent by editing the ambari-agent.ini file as shown below:
• vi /etc/ambari-agent/conf/ambari-agent.ini
  [server] hostname=
  url_port=8440
  secured_url_port=8441
3. Start the Agent on every host in your cluster.
• ambari-agent start
• The Agent registers with the Server on start.

The Agent should not die if the master suddenly disappears. It should continue to poll at regular intervals and recover as needed when the master comes back up:

The Ambari Agent should keep all the necessary information it planned to send to the master in case of a connection failure and re-send the information after the master comes back up. It may need to re-register if it was previously in the process of registering.

Troubleshooting

• The first thing to do if you run into trouble is to find the logs. Ambari Agent logs can be found at 
• /var/log/ambari-agent/ambari-agent.log

References

1. Non-Ambari Cluster Installation Guide (HDP)
2. Installing Hadoop Using Ambari
3. DEPLOYING, MANAGING AND CONFIGURING HDP WITH AMBARI 1.7 (tutorial)
4. Understanding the Basics
5. Ambari Architecture (pdf)
6. How can I start my Ambari heartbeat?
7. Installing Ambari Agents Manually
8. INTRODUCING APACHE AMBARI FOR DEPLOYING AND MANAGING APACHE HADOOP (HortonWorks)
9. Ambari Admin Guide (Version 2.2.2.0) 
10. Ambari Reference Guide (Version 2.2.2.0) 
11. Ambari User’s Guide (Version 2.2.2.0) 
12. Ambari Troubleshooting Guide (Version 2.2.2.0) 
13. Ambari Security Guide (Version 2.2.2.0) 
14. Automated Install with Ambari  (Version 2.2.2.0) 
15. Ambari Upgrade Guide (Version 2.2.2.0) 
16. Install, Configure, and Deploy an HDP Cluster 
17. Ambari Agent certificates (to be removed if you need to update the Agent)
• /var/lib/ambari-agent/keys/*
18. Blueprint Support for HA Clusters (Apache Ambari)
19. Ambari Metrics System ("AMS")
• A system for collecting, aggregating and serving Hadoop and system metrics in Ambari-managed clusters
20. All Cloud-related articles on Xml and More
21. Installing Spark Using Ambari (HDP-2.5.3)