These release notes cover new developer and user-facing incompatibilities, important issues, features, and major improvements.
Fixed security audit logger configuration. (Arpit Gupta via Eric Yang)
Enable task memory management to be configurable via hadoop config setup script.
Fixed hadoop-setup-conf.sh to put proxy user in core-site.xml. (Arpit Gupta via Eric Yang)
Added parameter for HBase user to setup config script. (Arpit Gupta via Eric Yang)
Removed unnecessary security logger configuration. (Eric Yang)
Fixed recursive sourcing of HADOOP_OPTS environment variables (Arpit Gupta via Eric Yang)
Fixed hadoop-setup-conf.sh to handle config file consistently. (Eric Yang)
Added toggle for dfs.support.append, webhdfs and hadoop proxy user to setup config script. (Arpit Gupta via Eric Yang)
Fixed conflict uid for install packages. (Eric Yang)
Added init.d script for jobhistory server and secondary namenode. (Eric Yang)
HADOOP-7681. Fixed security and hdfs audit log4j properties (Arpit Gupta via Eric Yang)
closing again
Committed to trunk and v23, since code reviewed by Eric.
Set hdfs uid, mapred uid, and hadoop gid to fixed numbers (201, 202, and 123, respectively).
WARNING: No release note provided for this incompatible change.
JVM metrics published to Ganglia now include the process name as part of the gmetric name.
Protocol buffer-generated types may now be used as arguments or return values for Hadoop RPC.
The scripts that run Hadoop no longer automatically add tools.jar from the JDK to the classpath (if it is present). If your job depends on tools.jar in the JDK you will need to add this dependency in your job.
hadoop-daemon.sh now returns a non-zero exit code if it detects that the daemon was not still running after 3 seconds.
Use of this new utility method avoids null result from File.listFiles(), and consequent NPEs.
Added missing library during creation of the eclipse project files.
The “Found X items” header on the output of the “du” command has been removed to more closely match unix. The displayed paths now correspond to the command line arguments instead of always being a fully qualified URI. For example, the output will have relative paths if the command line arguments are relative paths.
WARNING: No release note provided for this incompatible change.
viewfs - client-side mount table.
Updated the help for the touchz command.
MapWritable now implements equals() and hashCode() based on the map contents rather than object identity in order to correctly implement the Map interface.
Failmon removed from contrib codebase.
This is the COMMON portion of a fix requiring coordinated change of COMMON and HDFS. Please see HDFS-1445 for HDFS portion and release note.
Adding support for Kerberos HTTP SPNEGO authentication to the Hadoop web-consoles
Removed references to the older fs.checkpoint.* properties that resided in core-site.xml
Updates hadoop-config.sh to always resolve symlinks when determining HADOOP_HOME. Bash built-ins or POSIX:2001 compliant cmds are now required.
Adds __attribute__ ((unused))
Add a new API listCorruptFileBlocks to FIleContext that returns a list of files that have corrupt blocks.
Increments the RPC protocol version in org.apache.hadoop.ipc.Server from 4 to 5. Introduces ArrayPrimitiveWritable for a much more efficient wire format to transmit arrays of primitives over RPC. ObjectWritable uses the new writable for array of primitives for RPC and continues to use existing format for on-disk data.
Metrics names are standardized to CapitalizedCamelCase. See release note of HADOOP-6918 and HADOOP-6920.
Metrics names are standardized to use CapitalizedCamelCase. Some examples of this is:
New metrics2 framework for Hadoop.
WARNING: No release note provided for this incompatible change.
WARNING: No release note provided for this incompatible change.
Improve the buffer utilization of ZlibCompressor to avoid invoking a JNI per write request.
The native build run when from trunk now requires autotools, libtool and openssl dev libraries.
WARNING: No release note provided for this incompatible change.
The “rm” family of FsShell commands now supports -rmdir and -f options.
Added RPM/DEB packages to build system.
Removed dependency of testcase on /tmp and made it to use test.build.data directory instead.
Invalid configuration values now result in a number format exception rather than the default value being used.
HDFS now has the ability to use posix_fadvise and sync_data_range syscalls to manage the OS buffer cache. This support is currently considered experimental, and may be enabled by configuring the following keys: dfs.datanode.drop.cache.behind.writes - set to true to drop data out of the buffer cache after writing dfs.datanode.drop.cache.behind.reads - set to true to drop data out of the buffer cache when performing sequential reads dfs.datanode.sync.behind.writes - set to true to trigger dirty page writeback immediately after writing data dfs.datanode.readahead.bytes - set to a non-zero value to trigger readahead for sequential reads
This change allows when running multiple namenodes on different hosts, sharing the same configuration file across all the nodes in the cluster (Datanodes, NamNode, BackupNode, SecondaryNameNode), without the need to define dfs.federation.nameservice.id parameter.
Added a conf property dfs.webhdfs.enabled for enabling/disabling webhdfs.
Added two new conf properties dfs.web.authentication.kerberos.principal and dfs.web.authentication.kerberos.keytab for the SPNEGO servlet filter.
The hdfsproxy contrib component is no longer supported.
New dfsadmin command added: [-setBalancerBandwidth <bandwidth>] where bandwidth is max network bandwidth in bytes per second that the balancer is allowed to use on each datanode during balacing.
This is an incompatible change in 0.23. The versions of ClientProtocol and DatanodeProtocol are changed.
Moved block management codes to a new package org.apache.hadoop.hdfs.server.blockmanagement.
Declare methods in DataTransferProtocol interface, and change Sender and Receiver to implement the interface.
WARNING: No release note provided for this incompatible change.
WARNING: No release note provided for this incompatible change.
Add hdfsHFlush to libhdfs.
Added header classes for individual DataTransferProtocol op headers.
Create HDFS RPM package
Removed the deprecated fields in DataTransferProtocol.
* Removed duplicated jars in test class path.
Remove packaging of duplicated third party jar files
A multi-level mkdir is now POSIX compliant. Instead of creating intermediate directories with the permissions of the parent directory, intermediate directories are created with permission bits of rwxrwxrwx (0777) as modified by the current umask, plus write and search permission for the owner.
I have committed this. Thanks to Bharath!
I’ve committed this. Thanks, Matt!
Introduces a new command, “hdfs groups”, which displays what groups are associated with a user as seen by the NameNode.
Add a new DataTransferProtocol operation, Op.TRANSFER_BLOCK, for transferring RBW/Finalized with acknowledgement and without using RPC.
Updated the Quota exceptions to now use human readable output.
The masters file is no longer used to indicate which hosts to start the 2NN on. The 2NN is now started on hosts when dfs.namenode.secondary.http-address is configured with a non-wildcard IP.
Added a new stage TRANSFER_RBW to DataTransferProtocol
If dfs.name.dir points to an empty dir, namenode -format no longer requires confirmation.
Added a new configuration property dfs.block.invalidate.limit for FSNamesystem.blockInvalidateLimit.
Rename HdfsConstants interface to HdfsServerConstants, FSConstants interface to HdfsConstants
Added two configuration properties, dfs.client.block.write.replace-datanode-on-failure.enable and dfs.client.block.write.replace-datanode-on-failure.policy. Added a new feature to replace datanode on failure in DataTransferProtocol. Added getAdditionalDatanode(..) in ClientProtocol.
Removed references to the older fs.checkpoint.* properties that resided in core-site.xml
Implemented a daemon thread to monitor the disk usage for periodically and if the disk usage reaches the threshold value, put the name node into Safe mode so that no modification to file system will occur. Once the disk usage reaches below the threshold, name node will be put out of the safe mode. Here threshold value and interval to check the disk usage are configurable.
The native build run when from trunk now requires autotools, libtool and openssl dev libraries.
The permissions on datanode data directories (configured by dfs.datanode.data.dir.perm) now default to 0700. Upon startup, the datanode will automatically change the permissions to match the configured value.
Summary of changes to the decommissioning process:
On web UI, missing block number now becomes accurate and under-replicated blocks do not include missing blocks.
Make a client name has this format: DFSClient_applicationid_randomint_threadid, where applicationid = mapred.task.id or else = “NONMAPREDUCE”.
Offline edits viewer feature adds oev tool to hdfs script. Oev makes it possible to convert edits logs to/from native binary and XML formats. It uses the same framework as Offline image viewer.
Example usage:
$HADOOP_HOME/bin/hdfs oev -i edits -o output.xml
Batch hardlinking during “upgrade” snapshots, cutting time from aprx 8 minutes per volume to aprx 8 seconds. Validated in both Linux and Windows. Depends on prior integration with patch for HADOOP-7133.
Updated the JavaDocs to appropriately represent the new Configuration Keys that are used in the code. The docs did not match the code.
Added a check to match the sure RPC and HTTP Port’s on the NameNode were not set to the same value, otherwise an IOException is throw with the appropriate message.
Make the DataNode’s block-volume choosing policy pluggable.
Metrics names are standardized to use CapitalizedCamelCase. Some examples:
The NameNode’s storage layout for its name directories has been reorganized to be more robust. Each edit now has a unique transaction ID, and each file is associated with a transaction ID (for checkpoints) or a range of transaction IDs (for edit logs).
This changes the fsimage format to be root directory-1 directory-2 … directoy-n. Each directory stores all its children in the following format: Directory_full_path_name num_of_children child-1 … child-n. Each inode stores only the last component of its path name into fsimage. This change requires an upgrade at deployment.
Fixes Rumen to get correct hostName that includes rackName in attempt info.
Rumen is fixed to ignore the AMRestartedEvent.
Resource limits are now expressed and enforced in terms of physical memory, rather than virtual memory. The virtual memory limit is set as a configurable multiple of the physical limit. The NodeManager’s memory usage is now configured in units of MB rather than GB.
New Yarn configuration property:
Name: yarn.app.mapreduce.am.scheduler.connection.retries Description: Number of times AM should retry to contact RM if connection is lost.
Makes Rumen use job history api instead of relying on current history file name format.
Fixes TraceBuilder to handle 0.20 history file names also.
Removed inheritance of certain server environment variables (HADOOP_OPTS and HADOOP_ROOT_LOGGER) in task attempt process.
contrib/vaidya/bin/vaidya.sh script fixed to use appropriate jars and classpath
Simple typo fix to allow ResourceManager to start instead of fail
WARNING: No release note provided for this incompatible change.
Generate state graph from State Machine Definition
A new server has been added to yarn. It is a web proxy that sits in front of the AM web UI. The server is controlled by the yarn.web-proxy.address config. If that config is set, and it points to an address that is different then the RM web interface then a separate proxy server needs to be launched.
This can be done by running
yarn-daemon.sh start proxyserver
If a separate proxy server is needed other configs also may need to be set, if security is enabled. yarn.web-proxy.principal yarn.web-proxy.keytab
The proxy server is stateless and should be able to support a VIP or other load balancing sitting in front of multiple instances of this server.
Fixed a race condition in writing the log index file that caused tasks to ‘fail’.
“mapred/job -list” now contains map/reduce, container, and resource information.
Generalizes token renewal and canceling to a common interface and provides a plugin interface for adding renewers for new kinds of tokens. Hftp changed to store the tokens as HFTP and renew them over http.
The pre-MR2 MapReduce implementation (JobTracker, TaskTracer, etc) and contrib components are no longer supported. This implementation is currently supported in the 0.20.20x releases.
Adding a simple, DistributedShell application as an alternate framework to MapReduce and to act as an illustrative example for porting applications to YARN.
Enhance OutputCommitter and FileOutputCommitter to allow for recover of tasks across job restart.
New config added:
// the maximum number of completed applications the RM keeps <name>yarn.server.resourcemanager.expire.applications.completed.max</name>
IsolationRunner is no longer maintained. See MAPREDUCE-2637 for its replacement.
Gridmix now prints a summary information after every run. It summarizes the runs w.r.t input trace details, input data statistics, cli arguments, data-gen runtime, simulation runtimes etc and also the cluster w.r.t map slots, reduce slots, jobtracker-address, hdfs-address etc.
Adds system tests to test the High-Ram feature in Gridmix.
Adds distributed cache related system tests to Gridmix.
Adds system tests for testing the compression emulation feature of Gridmix.
Adds High-Ram feature emulation in Gridmix.
Added 2 new config parameters:
mapreduce.reduce.shuffle.catch.exception.stack.regex mapreduce.reduce.shuffle.catch.exception.message.regex
Created rpm and debian packages for MapReduce.
Adds system tests to Gridmix. These system tests cover various features like job types (load and sleep), user resolvers (round-robin, submitter-user, echo) and submission modes (stress, replay and serial).
Added config option mapreduce.tasktracker.cache.local.keep.pct to the TaskTracker. It is the target percentage of the local distributed cache that should be kept in between garbage collection runs. In practice it will delete unused distributed cache entries in LRU order until the size of the cache is less than mapreduce.tasktracker.cache.local.keep.pct of the maximum cache size. This is a floating point value between 0.0 and 1.0. The default is 0.95.
Map and Reduce task can access the attempt’s overall progress via TaskAttemptContext.
Removed duplicated hadoop-common library dependencies.
Looks great. I just committed this. Thanks Siddharth!
Improve the Partitioner interface’s docs to help fetch Job Configuration objects.
Introduces a new command, “mapred groups”, which displays what groups are associated with a user as seen by the JobTracker.
Task attempt’s total heap usage gets recorded and published via counters as COMMITTED_HEAP_BYTES.
WARNING: No release note provided for this incompatible change.
I just committed this. Thanks Luke!
MRUnit is now available as a separate Apache project.
Fixes Gridmix in RoundRobinUserResolver mode to map testing/proxy users to unique users in a trace.
Removes the restriction of specifying group names in users-list file for Gridmix in RoundRobinUserResolver mode.
Emulates the MapReduce compression feature in Gridmix. By default, compression emulation is turned on. Compression emulation can be disabled by setting ‘gridmix.compression-emulation.enable’ to ‘false’. Use ‘gridmix.compression-emulation.map-input.decompression-ratio’, ‘gridmix.compression-emulation.map-output.compression-ratio’ and ‘gridmix.compression-emulation.reduce-output.compression-ratio’ to configure the compression ratios at map input, map output and reduce output side respectively. Currently, compression ratios in the range [0.07, 0.68] are supported. Gridmix auto detects whether map-input, map output and reduce output should emulate compression based on original job’s compression related configuration parameters.
Makes Gridmix emulate HDFS based distributed cache files and local file system based distributed cache files.
An efficient implementation of small jobs by running all tasks in the MR ApplicationMaster JVM, there-by affecting lower latency.
The native build run when from trunk now requires autotools, libtool and openssl dev libraries.
TextInputFormat may now split lines with delimiters other than newline, by specifying a configuration parameter “textinputformat.record.delimiter”
MultipleOutputs should not require the use/check of ‘Writable’ interfaces in key and value classes.
Task-cleanup task should not be scheduled on the node that the task just failed
Adds job configuration parameters to the job trace. The configuration parameters are stored under the ‘jobProperties’ field as key-value pairs.
New configuration properties gridmix.job.original-job-id and gridmix.job.original-job-name in the configuration of simulated job are exposed/documented to gridmix user for mapping between original cluster’s jobs and simulated jobs.
Adds total heap usage emulation to Gridmix. Also, Gridmix can configure the simulated task’s JVM heap options with max heap options obtained from the original task (via Rumen). Use ‘gridmix.task.jvm-options.enable’ to disable the task max heap options configuration.
Adds cumulative cpu usage emulation to Gridmix
Adds cpu, physical memory, virtual memory and heap usages to TraceBuilder’s output.
Capture intermediate task resource consumption information: * Time taken so far * CPU load [either at the time the data are taken, or exponentially smoothed] * Memory load [also either at the time the data are taken, or exponentially smoothed]
This would be taken at intervals that depend on the task progress plateaus. For example, reducers have three progress ranges - [0-1/3], (1/3-2/3], and (2/3-3/3] - where fundamentally different activities happen. Mappers have different boundaries that are not symmetrically placed [0-9/10], (9/10-1]. Data capture boundaries should coincide with activity boundaries. For the state information capture [CPU and memory] we should average over the covered interval.
Fix a misleading documentation note about the usage of Reporter objects in Reducers.
Adds -recursive option to TraceBuilder for scanning the input directories recursively.
The default minimum heartbeat interval has been dropped from 3 seconds to 300ms to increase scheduling throughput on small clusters. Users may tune mapreduce.jobtracker.heartbeats.in.second to adjust this value.
Print the resultant status of a Job on completion instead of simply saying ‘Complete’.
WARNING: No release note provided for this incompatible change.
Added a ’‘-starts-after’ option to Rumen’s Folder utility. The time duration specified after the ‘-starts-after’ option is an offset with respect to the submit time of the first job in the input trace. Jobs in the input trace having a submit time (relative to the first job’s submit time) lesser than the specified offset will be ignored.
Fix a misleading exception message in case the Chained Mappers have mismatch in input/output Key/Value pairs between them.
Job names on jobtracker.jsp should be 80 characters long at most.
Added a helpful description message to the mapred job -events command.
Efficient implementation of MapReduce framework counters.
Fixed the streaming test TestStreamingExitStatus’s failure due to an OutOfMemory error by reducing the testcase’s io.sort.mb.
MapReduce has undergone a complete re-haul in hadoop-0.23 and we now have, what we call, MapReduce 2.0 (MRv2).
The fundamental idea of MRv2 is to split up the two major functionalities of the JobTracker, resource management and job scheduling/monitoring, into separate daemons. The idea is to have a global ResourceManager (RM) and per-application ApplicationMaster (AM). An application is either a single job in the classical sense of Map-Reduce jobs or a DAG of jobs. The ResourceManager and per-node slave, the NodeManager (NM), form the data-computation framework. The ResourceManager is the ultimate authority that arbitrates resources among all the applications in the system. The per-application ApplicationMaster is, in effect, a framework specific library and is tasked with negotiating resources from the ResourceManager and working with the NodeManager(s) to execute and monitor the tasks.
The ResourceManager has two main components: * Scheduler (S) * ApplicationsManager (ASM)
The Scheduler is responsible for allocating resources to the various running applications subject to familiar constraints of capacities, queues etc. The Scheduler is pure scheduler in the sense that it performs no monitoring or tracking of status for the application. Also, it offers no guarantees on restarting failed tasks either due to application failure or hardware failures. The Scheduler performs its scheduling function based the resource requirements of the applications; it does so based on the abstract notion of a Resource Container which incorporates elements such as memory, cpu, disk, network etc.
The Scheduler has a pluggable policy plug-in, which is responsible for partitioning the cluster resources among the various queues, applications etc. The current Map-Reduce schedulers such as the CapacityScheduler and the FairScheduler would be some examples of the plug-in.
The CapacityScheduler supports hierarchical queues to allow for more predictable sharing of cluster resources. The ApplicationsManager is responsible for accepting job-submissions, negotiating the first container for executing the application specific ApplicationMaster and provides the service for restarting the ApplicationMaster container on failure.
The NodeManager is the per-machine framework agent who is responsible for launching the applications’ containers, monitoring their resource usage (cpu, memory, disk, network) and reporting the same to the Scheduler.
The per-application ApplicationMaster has the responsibility of negotiating appropriate resource containers from the Scheduler, tracking their status and monitoring for progress.