51. What is Identity Mapper and Identity Reducer in MapReduce?
◦ org.apache.hadoop.mapred.lib.IdentityMapper: Implements the identity function, mapping inputs directly to outputs. If MapReduce programmer does not set the Mapper Class using JobConf.setMapperClass then IdentityMapper.class is used as a default value.
◦ org.apache.hadoop.mapred.lib.IdentityReducer: Performs no reduction, writing all input values directly to the output. If MapReduce programmer does not set the Reducer Class using JobConf.setReducerClass then IdentityReducer.class is used as a default value.
52. What is the meaning of speculative execution in Hadoop? Why is it important?
Ans: Speculative execution is a way of coping with individual Machine performance. In large clusters where hundreds or thousands of machines are involved there may be machines which are not performing as fast as others.
This may result in delays in a full job due to only one machine not performing well. To avoid this, speculative execution in Hadoop can run multiple copies of same map or reduce task on different slave nodes. The results from first node to finish are used
53. When the reducers are started in a MapReduce job?
Ans: In a MapReduce job reducers do not start executing the reduce method until the all Map jobs have completed. Reducers start copying intermediate key-value pairs from the mappers as soon as they are available. The programmer defined reduce method is called only after all the mappers have finished.
If reducers do not start before all mappers finish then why does the progress on MapReduce job shows something like Map (50%) Reduce (10%)? Why reducer’s progress percentage is displayed when mapper is not finished yet?
Reducers start copying intermediate key-value pairs from the mappers as soon as they are available. The progress calculation also takes in account the processing of data transfer which is done by reduce process, therefore the reduce progress starts showing up as soon as any intermediate key-value pair for a mapper is available to be transferred to reducer.
Though the reducer progress is updated still the programmer defined reduce method is called only after all the mappers have finished.
54. What is HDFS? How it is different from traditional file systems?
Ans: HDFS, the Hadoop Distributed File System, is responsible for storing huge data on the cluster. This is a distributed file system designed to run on commodity hardware. It has many similarities with existing distributed file systems. However, the differences from other distributed file systems are significant.
◦ HDFS is highly fault-tolerant and is designed to be deployed on low-cost hardware.
◦ HDFS provides high throughput access to application data and is suitable for applications that have large data sets.
◦ HDFS is designed to support very large files. Applications that are compatible with HDFS are those that deal with large data sets. These applications write their data only once but they read it one or more times and require these reads to be satisfied at streaming speeds. HDFS supports write-once- read-many semantics on files.
55. What is HDFS Block size? How is it different from traditional file system block size?
Ans: In HDFS data is split into blocks and distributed across multiple nodes in the cluster. Each block is typically 64Mb or 128Mb in size. Each block is replicated multiple times. Default is to replicate each block three times. Replicas are stored on different nodes. HDFS utilizes the local file system to store each HDFS block as a separate file. HDFS Block size cannot be compared with the traditional file system block size.
57. What is a Name Node? How many instances of Name Node run on a Hadoop Cluster?
Ans: The Name Node is the centrepiece of an HDFS file system. It keeps the directory tree of all files in the file system, and tracks where across the cluster the file data is kept. It does not store the data of these files itself. There is only One Name Node process run on any Hadoop cluster. Name Node runs on its own JVM process. In a typical production cluster its run on a separate machine. The Name Node is a Single Point of Failure for the HDFS Cluster. When the Name Node goes down, the file system goes offline.
Client applications talk to the Name Node whenever they wish to locate a file, or when they want to add/copy/move/delete a file. The Name Node responds the successful requests by returning a list of relevant Data Node servers where the data lives.
58. What is a Data Node? How many instances of Data Node run on a Hadoop Cluster?
Ans: A Data Node stores data in the Hadoop File System HDFS. There is only One Data Node process run on any Hadoop slave node. Data Node runs on its own JVM process. On startup, a Data Node connects to the Name Node. Data Node instances can talk to each other, this is mostly during replicating data.
59. How the Client communicates with HDFS?
Ans: The Client communication to HDFS happens to be using Hadoop HDFS API. Client applications talk to the Name Node whenever they wish to locate a file, or when they want to add/copy/move/delete a file on HDFS. The Name Node responds the successful requests by returning a list of relevant Data Node servers where the data lives. Client applications can talk directly to a Data Node, once the Name Node has provided the location of the data.
60. How the HDFS Blocks are replicated?
Ans: HDFS is designed to reliably store very large files across machines in a large cluster. It stores each
file as a sequence of blocks; all blocks in a file except the last block are the same size.
The blocks of a file are replicated for fault tolerance. The block size and replication factor are configurable per file. An application can specify the number of replicas of a file. The replication factor can be specified at file creation time and can be changed later. Files in HDFS are writing-once and have strictly one writer at any time.