Apache Hadoop 2.1.0-beta Release Notes

These release notes cover new developer and user-facing incompatibilities, important issues, features, and major improvements.


Client ping will be sent as a RPC header with a reserved callId instead of as a sentinel RPC packet length.


WARNING: No release note provided for this incompatible change.


The RPC client now waits for the Server’s SASL negotiate response before instantiating its SASL client.


Connection context is now sent as a rpc header wrapped protobuf.


WARNING: No release note provided for this incompatible change.


WARNING: No release note provided for this incompatible change.


WARNING: No release note provided for this incompatible change.


Raw SASL protocol now uses protobufs wrapped with RPC headers. The negotiation sequence incorporates the state of the exchange. The server now has the ability to advertise its supported auth types.


WARNING: No release note provided for this incompatible change.


fixed in HADOOP-9258


fixed in HADOOP-9258


Part of the RPC version 9 change. A service class byte is added after the version byte.


WARNING: No release note provided for this incompatible change.


WARNING: No release note provided for this incompatible change.


Kosmos FS (KFS) is no longer maintained and Hadoop support has been removed. KFS has been replaced by QFS (HADOOP-8885).


This umbrella jira makes enhancements to support Hadoop natively on Windows Server and Windows Azure environments.


This patch should be checked in together (or after) with JIRA Hadoop-8469: https://issues.apache.org/jira/browse/HADOOP-8469


WARNING: No release note provided for this incompatible change.


The dfsadmin -metasave command has been changed to overwrite the output file. Previously, this command would append to the output file if it already existed.


The Protocol Buffers definition of the inter-namenode protocol required a change for compatibility with compiled C clients. This is a backwards-incompatible change. A namenode prior to this change will not be able to communicate with a namenode after this change.


This change makes name node keep its internal replication queues and data node state updated in manual safe mode. This allows metrics and UI to present up-to-date information while in safe mode. The behavior during start-up safe mode is unchanged.


WARNING: No release note provided for this incompatible change.


With this improvement the following options are available in release 1.2.0 and later on 1.x release stream: 1. jsvc location can be overridden by setting environment variable JSVC_HOME. Defaults to jsvc binary packaged within the Hadoop distro. 2. jsvc log output is directed to the file defined by JSVC_OUTFILE. Defaults to $HADOOP_LOG_DIR/jsvc.out. 3. jsvc error output is directed to the file defined by JSVC_ERRFILE file. Defaults to $HADOOP_LOG_DIR/jsvc.err.

With this improvement the following options are available in release 2.0.4 and later on 2.x release stream: 1. jsvc log output is directed to the file defined by JSVC_OUTFILE. Defaults to $HADOOP_LOG_DIR/jsvc.out. 2. jsvc error output is directed to the file defined by JSVC_ERRFILE file. Defaults to $HADOOP_LOG_DIR/jsvc.err.

For overriding jsvc location on 2.x releases, here is the release notes from HDFS-2303: To run secure Datanodes users must install jsvc for their platform and set JSVC_HOME to point to the location of jsvc in their environment.


This change adds support for referencing files and directories based on fileID/inodeID using a path /.reserved/.inodes/<inodeid>. With this change creating a file or directory /.reserved is not longer allowed. Before upgrading to a release with this change, files /.reserved needs to be renamed to another name.


This change introduces a maximum number of blocks per file, by default one million, and a minimum block size, by default 1MB. These can optionally be changed via the configuration settings “dfs.namenode.fs-limits.max-blocks-per-file” and “dfs.namenode.fs-limits.min-block-size”, respectively.


The default blocks size prior to this change was 64MB. This jira changes the default block size to 128MB. To go back to previous behavior, please configure the in hdfs-site.xml, the configuration parameter “dfs.blocksize” to 67108864.


WARNING: No release note provided for this incompatible change.


There is now a new option to have the DN take into account available disk space on each volume when choosing where to place a replica when performing an HDFS write. This can be enabled by setting the config “dfs.datanode.fsdataset.volume.choosing.policy” to the value “org.apache.hadoop.hdfs.server.datanode.fsdataset.AvailableSpaceVolumeChoosingPolicy”.


Fixes blank Configuration object creation overhead by reusing the Job configuration in InMemoryReader.


WARNING: No release note provided for this incompatible change.


WARNING: No release note provided for this incompatible change.


WARNING: No release note provided for this incompatible change.


WARNING: No release note provided for this incompatible change.


WARNING: No release note provided for this incompatible change.


Document MR Binary Compatibility vis-a-vis hadoop-1 and hadoop-2 for end-users.


WARNING: No release note provided for this incompatible change.


WARNING: No release note provided for this incompatible change.


WARNING: No release note provided for this incompatible change.


Ensure that mapreduce APIs are semantically consistent with mapred API w.r.t Mapper.cleanup and Reducer.cleanup; in the sense that cleanup is now called even if there is an error. The old mapred API already ensures that Mapper.close and Reducer.close are invoked during error handling. Note that it is an incompatible change, however end-users can override Mapper.run and Reducer.run to get the old (inconsistent) behaviour.


Made the method ParsedTask.obtainTaskAttempts() public.


Fixes Rumen to parse counter strings containing the special characters “{” and “}”.


Bug fixed in compression emulation feature for map only jobs.


Fixes NPE in cpu emulation in Gridmix


WARNING: No release note provided for this incompatible change.


Fixes NPE and makes Gridmix simulate succeeded-jobs-with-failed-tasks. All tasks of such simulated jobs(including the failed ones of original job) will succeed.


Makes Gridmix emit out correct error message when the input data directory already exists and -generate option is used. Makes Gridmix exit with proper exit codes when Gridmix fails in args-processing, startup/setup.


JobMonitor can now deploy multiple threads for faster job-status polling. Use ‘gridmix.job-monitor.thread-count’ to set the number of threads. Stress mode now relies on the updates from the job monitor instead of polling for job status. Failures in job submission now get reported to the statistics module and ultimately reported to the user via summary.


Fixed the sortFinishTime and shuffleFinishTime adjustments in Rumen Folder.


Makes Gridmix use the uncompressed input data size while simulating map tasks in the case where compressed input data was used in original job.


WARNING: No release note provided for this incompatible change.


WARNING: No release note provided for this incompatible change.


WARNING: No release note provided for this incompatible change.


WARNING: No release note provided for this incompatible change.


WARNING: No release note provided for this incompatible change.


WARNING: No release note provided for this incompatible change.


WARNING: No release note provided for this incompatible change.


WARNING: No release note provided for this incompatible change.


WARNING: No release note provided for this incompatible change.


WARNING: No release note provided for this incompatible change.


WARNING: No release note provided for this incompatible change.


WARNING: No release note provided for this incompatible change.


WARNING: No release note provided for this incompatible change.


WARNING: No release note provided for this incompatible change.


WARNING: No release note provided for this incompatible change.


WARNING: No release note provided for this incompatible change.


WARNING: No release note provided for this incompatible change.


WARNING: No release note provided for this incompatible change.


WARNING: No release note provided for this incompatible change.


WARNING: No release note provided for this incompatible change.


WARNING: No release note provided for this incompatible change.


WARNING: No release note provided for this incompatible change.


WARNING: No release note provided for this incompatible change.


WARNING: No release note provided for this incompatible change.


WARNING: No release note provided for this incompatible change.


WARNING: No release note provided for this incompatible change.


WARNING: No release note provided for this incompatible change.


WARNING: No release note provided for this incompatible change.


WARNING: No release note provided for this incompatible change.


WARNING: No release note provided for this incompatible change.


WARNING: No release note provided for this incompatible change.


WARNING: No release note provided for this incompatible change.


WARNING: No release note provided for this incompatible change.


WARNING: No release note provided for this incompatible change.


WARNING: No release note provided for this incompatible change.


WARNING: No release note provided for this incompatible change.


WARNING: No release note provided for this incompatible change.


WARNING: No release note provided for this incompatible change.


WARNING: No release note provided for this incompatible change.


WARNING: No release note provided for this incompatible change.


WARNING: No release note provided for this incompatible change.


WARNING: No release note provided for this incompatible change.


WARNING: No release note provided for this incompatible change.


WARNING: No release note provided for this incompatible change.


WARNING: No release note provided for this incompatible change.


WARNING: No release note provided for this incompatible change.


WARNING: No release note provided for this incompatible change.


WARNING: No release note provided for this incompatible change.


WARNING: No release note provided for this incompatible change.


WARNING: No release note provided for this incompatible change.


WARNING: No release note provided for this incompatible change.


WARNING: No release note provided for this incompatible change.


WARNING: No release note provided for this incompatible change.


WARNING: No release note provided for this incompatible change.