These release notes cover new developer and user-facing incompatibilities, important issues, features, and major improvements.
Support to parallel building with maven command, such as `mvn -T 8 clean package -DskipTests`.
S3 Select is no longer supported through the S3A connector
If the user wants to load custom implementations of AWS Credential Providers through user provided jars can set {{fs.s3a.extensions.isolated.classloader}} to {{false}}.
maven/ivy imports of hadoop-common are less likely to end up with log4j versions on their classpath.
PositionedReadable.readVectored() will read incorrect data when reading from hdfs, azure abfs and other stores when given a direct buffer allocator.
For cross-version compatibility, use on-heap buffer allocators only
Drop support for Hbase V1 as the back end of the YARN Application Timeline service, which becomes HBase 2 only. The supported version HBase version is 2.5.8
This does not have any effect on HBase deployments themselves
Added a new conf “hadoop.security.crypto.jce.provider.auto-add” (default: true) to enable/disable auto-adding BouncyCastleProvider. This change also avoid statically loading the BouncyCastleProvider class.
Adds new Mount Table mode: LEADER_FOLLOWER for RBF
Apache httpclient 4.5.x is a new implementation of http connections; this supports a large configurable pool of connections along with the ability to limit their lifespan.
The networking library can be chosen using the configuration option fs.azure.networking.library
The supported values are - JDK_HTTP_URL_CONNECTION : Use JDK networking library [Default] - APACHE_HTTP_CLIENT : Use Apache HttpClient
Important: when the networking library is switched to the Apache http client, the apache httpcore and httpclient must be on the classpath.
hadoop modules no longer export protobuf-2.5.0 as a dependency, and it is omitted from the hadoop distribution directory. Applications which use the library must declare an explicit dependency.
Hadoop uses a shaded version of protobuf3 internally, and does not use the 2.5.0 JAR except when compiling compatible classes. It is still included in the binary distributions when the yarn timeline server is built with hbase 1
S3A upload operations can now recover from failures where the store returns a 500 error. There is an option to control whether or not the S3A client itself attempts to retry on a 50x error other than 503 throttling events (which are independently processed as before). Option: fs.s3a.retry.http.5xx.errors . Default: true
protobuf-2.5.jar is no longer bundled into a distribution or exported as a transient dependency.
Hadoop has upgraded to commons-collections4-4.4. This MUST be on the classpath to create a Configuration() object. The hadoop-commons dependency exports has been updated appropriately. If dependencies are configured manually, these MUST be updated. Applications which require the older “commons-collections” binaries on their classpath may have to explicitly add them.
Backwards-incompatible upgrade for security reasons. All field access is now via setter/getter methods. To marshal Serializable objects the packages they are in must be declared in the system property “{{org.apache.avro.SERIALIZABLE_PACKAGES}}”
This upgrade does break our compatibility policy -but this is a critical security issue. Everyone using an avro version before 1.11.4 MUST upgrade.
WARNING: No release note provided for this change.
NN fine-grained locking (FGL) implementation aims to alleviate the bottleneck by allowing concurrency of disjoint write operations. This is the phase one works. See HDFS-17384 for more information on this feature.
Asynchronous router RPC is designed to address the performance bottlenecks of synchronous router RPC in scenarios with high concurrency and multiple named services. By introducing an asynchronous processing mechanism, it optimizes the request-handling process, enhances the system’s concurrency capacity and resource utilization efficiency.
See the Apache JIRA ticket HDFS-17531 for more information on this feature. Further improvements are in the Apache JIRA ticket HDFS-17716.
Jetty has been upgraded to address CVE-2024-22201, CVE-2023-44487, CVE-2024-8184, CVE-2024-13009
The option fs.s3a.create.checksum.algorithm allows checksums to be set on file upload; It supports the following values: ‘CRC32’, ‘CRC32C’, ‘SHA1’, and ‘SHA256’. A future release will also support “none”.
A Hadoop 2.10.1 or 2.10.2 client connecting to a Hadoop 3.4.0~3.4.1, 3.3.0~3.3.6, 3.2.1~3.2.4 (any version with HDFS-13541) cluster could cause the DataNode to disconnect any subsequent client connections due to an incompatible binary protocol change.
HDFS-16644 provides a partial fix: a Hadoop 3 cluster will reject Hadoop 2.10.1/2.10.2 clients, but it will not fail other subsequent client connections.
For Hadoop 2 cluster wishing to upgrade to Hadoop 3 in a rolling fashion, the workaround is to perform a two-step upgrade: upgrade to an earlier Hadoop 3 version without HDFS-13541, and then upgrade again to the newer Hadoop 3 version. Or revert HDFS-13541 from your version and rebuild.
S3A client now uses S3 conditional overwrite PUT requests to perform overwrite protection checks at end of PUT request (stream close()). This saves a HEAD request on file creation, and actually delivers an atomic creation. It may not be supported on third party stores: set fs.s3a.create.conditional.enabled to false to revert to the old behavior. Consult the third-party-stores documentation for details.
AWS SDK 2.30.0 and later are (currently) incompatible with third party stores. Accordingly, this release is kept at a 2.29 version. See HADOOP-19490. There may now be some problems using the AWS4SignerType and S3.
S3A output streams no longer log warnings on use of hflush()/raise exceptions if fs.s3a.downgrade.syncable.exceptions = false. hsync() is reported with a warning/rejected, as appropriate. That method us absolutely unsupported when writing to S3
Option “fs.file.checksum.verify” disables checksum verification in local FS, so sliced subsets of larger buffers are never returned.
Stream capability “fs.capability.vectoredio.sliced” is true if a filesystem knows that it is returning slices of a larger buffer. This is false if a filesystem doesn’t, or against the local FS in releases which lack this feature.
Apache Hadoop now includes a FileSystem implementation backed by Google Cloud Storage (GCS), accessible through the gs:// URI scheme.
Previously, Hadoop has set the guava dependency version to 27.0-jre. It is now set to 32.0.1-jre . Note that this only applies to the un-relocated guava library, the Hadoop thirdparty guava library was already at 32.0.1-jre.
Wildfly SSL binding are now compatible with OpenSSL 3.0 as well as 1.1. However, as the native libraries were built with GLIBC 2.34+, these do not work on RHEL8. For deployment on those systems, stick with the JVM ssl support
Checksum calculation in the AWS SDK has changed and may be incompatible with third party stores. Consult the third-party documentation for details on restoring compatibility with the options “fs.s3a.md5.header.enabled” and “fs.s3a.checksum.calculation.enabled”
The hadoop-* modules needed to communicate with cloud stores (hadoop-azure, hadoop-aws etc) are now in share/hadoop/tools/lib, and will be automatically available on the command line provided all their dependencies are in the same directory. For hadoop-aws, that means aws sdk bundle.jar 2.35.4 or later; for hadoop-azure everything should work out the box.
Introduced a new modern UI for monitoring the Capacity Scheduler and dynamically editing its configuration.
The hadoop Lz4Compressor instantiates a compressor via a call to LZ4Factory.fastestInstance().safeDecompressor() and so is not directly vulnerable to CVE-2025-12183
The new Capacity Scheduler configuration UI is now served under rm-host:port/scheduler-ui
People testing timeline server integration through the FileSystemTimelineReaderImpl implementation of ATSv2 need to add hadoop-yarn-server-timelineservice-test.jar to their testing classpath
WASB Driver Client Is removed from official hadoop releases starting from hadoop-3.5.0.
Moving ahead ABFS Driver will be the only official Hadoop connector for both types of Accounts that is “Heirarchichal Namespace Enabled (HNS)” and “Heirarchichal Namespace Disabled (FNS)”
For more details refer to: https://github.com/apache/hadoop/blob/trunk/hadoop-tools/hadoop-azure/src/site/markdown/wasb.md
Introduced a modern UI for updating and managing the YARN Capacity Scheduler configuration.