Apache Hadoop 3.3.4 incorporates a number of significant enhancements over the previous major release line (hadoop-3.2).
Users are encouraged to read the full set of release notes. This page provides an overview of the major changes.
Protobuf upgraded to 3.7.1 as protobuf-2.5.0 reached EOL.
Hadoop now switches to use a shaded version of Guava from hadoop-thirdparty which helps to remove Guava version conflicts with downstream applications.
For LZ4 and Snappy compression codec, Hadoop now moves to use lz4-java and snappy-java instead of requring the native libraries of these to be installed on the systems running Hadoop.
External services or YARN service may need to call into WebHDFS or YARN REST API on behave of the user using web protocols. It would be good to support impersonation mechanism in AuthenticationFilter or similar extensions.
Lots of enhancements to the S3A code including Delegation Token support, better handling of 404 caching, S3guard performance, resilience improvements
Address issues which surface in the field and tune things which need tuning, add more tests where appropriate. Improve docs, especially troubleshooting.
HDFS Router now supports security. Also contains many bug fixes and improvements.
Aims to enable storage class memory first in read cache. Although storage class memory has non-volatile characteristics, to keep the same behavior as current read only cache, we don’t use its persistent characteristics currently.
application catalog system which provides an editorial and search interface for YARN applications. This improves usability of YARN for manage the life cycle of applications.
Tencent cloud is top 2 cloud vendors in China market and the object store COS is widely used among China’s cloud users. This task implements a COSN filesytem to support Tencent cloud COS natively in Hadoop.
scheduling of opportunistic container through the central RM (YARN-5220), through distributed scheduling (YARN-2877), as well as the scheduling of containers based on actual node utilization (YARN-1011) and the container promotion/demotion (YARN-5085).
The Hadoop documentation includes the information you need to get started using Hadoop. Begin with the Single Node Setup which shows you how to set up a single-node Hadoop installation. Then move on to the Cluster Setup to learn how to set up a multi-node Hadoop installation.