Apache Hadoop Changelog

Release 3.4.2 - 2025-08-20

IMPORTANT ISSUES:

JIRA Summary Priority Component Reporter Contributor
HADOOP-19107 Drop support for HBase v1 timeline service & upgrade HBase v2 Major . Ayush Saxena Ayush Saxena

NEW FEATURES:

JIRA Summary Priority Component Reporter Contributor
HADOOP-19261 Support force close a DomainSocket for server service Major . Sammi Chen Sammi Chen
HDFS-17657 The balancer service supports httpserver. Minor balancer & mover Zhaobo Huang Zhaobo Huang
HADOOP-19353 Über-jira: S3A Hadoop 3.4.2 features Major fs/s3 Steve Loughran Steve Loughran
HADOOP-19363 S3A: Support analytics-accelerator-s3 input streams for parquet read performance Major fs/s3 Ahmar Suhail Ahmar Suhail

IMPROVEMENTS:

JIRA Summary Priority Component Reporter Contributor
HADOOP-19195 Upgrade aws sdk v2 to 2.25.53 Major fs/s3 Harshit Gupta Harshit Gupta
YARN-11711 Improve ServiceScheduler Code Minor scheduler, yarn Shilun Fan Shilun Fan
HADOOP-15760 Upgrade commons-collections to commons-collections4 Major . David Mollitor Nihal Jain
HADOOP-19281 MetricsSystemImpl should not print INFO message in CLI Major metrics Tsz-wo Sze Sarveksha Yeshavantha Raju
YARN-11734 Fix spotbugs in ServiceScheduler#load Major yarn Hualong Zhang Hualong Zhang
HADOOP-19349 S3A : Improve Client Side Encryption Documentation Major documentation, fs/s3 Syed Shameerur Rahman Syed Shameerur Rahman
HADOOP-19357 ABFS: Optimizations for Retry Handling and Throttling Minor fs/azure Manika Joshi Manika Joshi
HADOOP-19366 Install OpenJDk 17 in default ubuntu build container Major build Cheng Pan Cheng Pan
HADOOP-19371 JVM GC Metrics supports ZGC pause time and count Minor common dzcxzl dzcxzl
HDFS-17711 Change fsimage loading progress percentage discontinuous to continuous Minor . Sungdong Kim Sungdong Kim
HADOOP-19389 Optimize shell -text command I/O with multi-byte read. Minor command, fs, fs/azure, fs/gcs, fs/s3 Chris Nauroth Chris Nauroth
HADOOP-19401 Improve error message when OS can’t identify the current user. Major security Chris Nauroth Chris Nauroth
HADOOP-19375 Organize JDK version-specific code in IDEA friendly approach Major . Cheng Pan Cheng Pan
HADOOP-19377 Avoid initializing useless HashMap in protocolImplMapArray. Minor common farmmamba farmmamba
HADOOP-19086 Update commons-logging to 1.3.0 Minor build Steve Loughran Steve Loughran
HDFS-17754 Add uriparser2 to notices Minor hdfs-client Chris Nauroth Chris Nauroth
HADOOP-19400 Expand specification and contract test coverage for InputStream reads. Major documentation, fs, test Chris Nauroth Chris Nauroth
HADOOP-19225 Upgrade Jetty to 9.4.57.v20241219 due to CVE-2024-8184 and other CVEs Major build Palakur Eshwitha Sai PJ Fanning
HADOOP-18991 Remove commons-beanutils dependency from Hadoop 3 Major common Istvan Toth Istvan Toth
HADOOP-19546 Include cipher feature for HttpServer2 and SSLFactory Major hadoop-common, hdfs, yarn Bence Kosztolnik Bence Kosztolnik
HDFS-17226 Building native libraries fails on Fedora 38 Major libhdfs++, native Kengo Seki Kengo Seki
HADOOP-19523 Upgrade to hadoop-thirdparty 1.4.0 Major build Steve Loughran Steve Loughran
HDFS-17637 Fix spotbugs in HttpFSFileSystem#getXAttr Major httpfs Hualong Zhang Hualong Zhang
HADOOP-19384 S3A: Add support for ProfileCredentialsProvider Minor fs/s3 Venkatasubrahmanian Narayanan Venkatasubrahmanian Narayanan
HADOOP-19143 Upgrade commons-cli to 1.6.0. Major build, common Shilun Fan Shilun Fan

BUG FIXES:

JIRA Summary Priority Component Reporter Contributor
HDFS-17376 Distcp creates Factor 1 replication file on target if Source is EC Major distcp Sadanand Shenoy Sadanand Shenoy
HDFS-16084 getJNIEnv() returns invalid pointer when called twice after getGlobalJNIEnv() failed Major libhdfs Antoine Pitrou kevin cai
HADOOP-19248 Protobuf code generate and replace should happen together Major common Cheng Pan Cheng Pan
HADOOP-19250 Fix test TestServiceInterruptHandling.testRegisterAndRaise Major test Chenyu Zheng Chenyu Zheng
HADOOP-19272 S3A: AWS SDK 2.25.53 warnings logged about transfer manager not using CRT client Major fs/s3 Steve Loughran Steve Loughran
HADOOP-19164 Hadoop CLI MiniCluster is broken Major . Ayush Saxena Ayush Saxena
HADOOP-19290 Operating on / in ChecksumFileSystem throws NPE Major . Ayush Saxena Ayush Saxena
HADOOP-19280 ABFS: Initialize ABFS client timer only when metric collection is enabled Major fs/azure Manish Bhatt Manish Bhatt
YARN-11732 Potential NPE when calling SchedulerNode#reservedContainer for CapacityScheduler Major capacityscheduler Tao Yang Tao Yang
HADOOP-19309 S3A CopyFromLocalFile operation fails when the source file does not contain file scheme. Major fs/s3 Syed Shameerur Rahman Syed Shameerur Rahman
HADOOP-18583 hadoop checknative fails to load openssl 3.x Major native Sebastian Klemke Sebastian Klemke
HADOOP-19106 [ABFS] All tests of. ITestAzureBlobFileSystemAuthorization fails with NPE Major fs/azure Mukund Thakur Anuj Modi
HADOOP-19360 Disable releases for apache.snapshots repo Major build Attila Doroszlai Attila Doroszlai
HDFS-17706 TestBlockTokenWithDFSStriped fails due to closed streams Minor . Felix N Felix N
YARN-11759 Fix log statement in RMAppImpl#processNodeUpdate Major resourcemanager yang yang yang yang
HDFS-17724 Set recover.lease.on.close.exception as an instance member in the DfsClientConf.java Minor hadoop-client Abhey Rana Abhey Rana
YARN-11745 YARN ResourceManager throws java.lang.IllegalArgumentExceptio: Comparison method violates its general contract! Major yarn chhinlinghean chhinlinghean
YARN-11785 Race condition in QueueMetrics due to non-thread-safe HashMap causes MetricsException Major capacityscheduler Tao Yang Tao Yang
YARN-11776 Handle NPE in the RMDelegationTokenIdentifier if localServiceAddress is null Major resourcemanager Abhey Rana Abhey Rana
HDFS-17759 Explicitly depend on jackson-core in hadoop-hdfs Critical hdfs Istvan Toth Istvan Toth
HADOOP-19488 RunJar throws UnsupportedOperationException on Windows Major hadoop-common Sangjin Lee Sangjin Lee
HDFS-16644 java.io.IOException Invalid token in javax.security.sasl.qop Major . Walter Su Zilong Zhu
HDFS-15230 Sanity check should not assume key base name can be derived from version name Major . Wei-Chiu Chuang Wei-Chiu Chuang
YARN-11810 Fix SQL script in SQLServer/FederationStateStoreTables.sql Major federation Peter Szucs Peter Szucs
HDFS-17768 Observer namenode network delay causing empty block location for getBatchedListing Major namenode Dimas Shidqi Parikesit Dimas Shidqi Parikesit
HADOOP-19532 Update commons-lang3 to 3.17.0 Major . Istvan Toth Istvan Toth
HADOOP-19551 Fix compilation error of native libraries on newer GCC Major native Masatake Iwasaki Masatake Iwasaki
YARN-11801 NPE in FifoCandidatesSelector.selectCandidates when preempting resources for an auto-created queue without child queues Major yarn Susheel Gupta Susheel Gupta
HADOOP-19554 LocalDirAllocator still doesn’t always recover from directory tree deletion Major common Steve Loughran Steve Loughran
YARN-11713 yarn-ui build fails in ARM docker in MacOs phantomjs error Blocker yarn-ui-v2 Mukund Thakur Masatake Iwasaki
YARN-11712 Yarn-ui build fails in ARM docker looking for python2. Blocker yarn-ui-v2 Mukund Thakur Masatake Iwasaki
HADOOP-19576 Insert Overwrite Jobs With MagicCommitter Fails On S3 Express Storage Major . Syed Shameerur Rahman Syed Shameerur Rahman
HADOOP-19573 S3A: ITestS3AConfiguration.testDirectoryAllocatorDefval() failing Minor fs/s3, test Steve Loughran Steve Loughran

TESTS:

JIRA Summary Priority Component Reporter Contributor
YARN-11744 Tackle flaky test testGetRunningContainersToKill Major test, yarn-service Cheng Pan Cheng Pan
YARN-11790 TestAmFilter#testProxyUpdate fails in some networks Minor test, webproxy Chris Nauroth Chris Nauroth

SUB-TASKS:

JIRA Summary Priority Component Reporter Contributor
HADOOP-18325 ABFS: Add correlated metric support for ABFS operations Major fs/azure Anmol Asrani Anmol Asrani
HADOOP-19187 ABFS: [FnsOverBlob] Making AbfsClient Abstract for supporting both DFS and Blob Endpoint Major fs/azure Anuj Modi Anuj Modi
HADOOP-19262 [JDK17] Upgade wildfly-openssl:1.1.3.Final to 2.1.4.Final+ Major fs/azure, fs/s3 Steve Loughran Saikat Roy
HADOOP-19284 ABFS: Allow “fs.azure.account.hns.enabled” to be set as Account Specific Config Major fs/azure Descifrado Anuj Modi
HADOOP-19219 Resolve Certificate error in Hadoop-auth tests. Major build, common Muskan Mishra Muskan Mishra
HADOOP-19296 [JDK17] Upgrade maven-war-plugin to 3.4.0 Major build, common Shilun Fan Shilun Fan
HADOOP-19286 Support S3A cross region access when S3 region/endpoint is set Major fs/s3 Syed Shameerur Rahman Syed Shameerur Rahman
HADOOP-19310 [JDK17] Add JPMS options required by Java 17+ Major build, common Cheng Pan Cheng Pan
HADOOP-19297 [JDK17] Upgrade maven.plugin-tools.version to 3.10.2 Major build, common Shilun Fan Shilun Fan
HADOOP-19330 S3A: Add LeakReporter; use in S3AInputStream Major fs/s3 Steve Loughran Steve Loughran
HADOOP-18708 AWS SDK V2 - Implement CSE Major fs/s3 Ahmar Suhail Syed Shameerur Rahman
HADOOP-19317 S3A: fs.s3a.connection.expect.continue controls 100 CONTINUE behavior Major fs/s3 Steve Loughran Steve Loughran
HADOOP-18960 ABFS contract-tests with Hadoop-Commons intermittently failing Minor fs/azure Pranav Saxena Anuj Modi
HADOOP-19336 S3A: Test failures after CSE support added Major fs/s3 Steve Loughran Syed Shameerur Rahman
HADOOP-19226 ABFS: [FnsOverBlob] Implementing Azure Rest APIs on Blob Endpoint for AbfsBlobClient Major fs/azure Anuj Modi Anuj Modi
HADOOP-19367 Fix setting final field value on Java 17 Major test Cheng Pan Cheng Pan
HADOOP-19207 ABFS: [FnsOverBlob] Response Handling of Blob Endpoint APIs and Metadata APIs Major fs/azure Anuj Modi Anuj Modi
HADOOP-19229 Vector IO on cloud storage: what is a good minimum seek size? Major fs/s3 Steve Loughran Steve Loughran
HADOOP-19393 ABFS: Returning FileAlreadyExists Exception for UnauthorizedBlobOverwrite Rename Errors Minor fs/azure Manika Joshi Manika Joshi
HADOOP-19232 ABFS: [FnsOverBlob] Implementing Ingress Support with various Fallback Handling Major fs/azure Anuj Modi Anmol Asrani
HADOOP-19233 ABFS: [FnsOverBlob] Implementing Rename and Delete APIs over Blob Endpoint Major fs/azure Anuj Modi Manish Bhatt
HADOOP-19404 ABFS: [FNS Over Blob] Update documentation for FNS Blob Onboard Major fs/azure Anmol Asrani Anmol Asrani
HADOOP-19448 ABFS: [FnsOverBlob][Optimizations] Reduce Network Calls In Create and Mkdir Flow Major fs/azure Anuj Modi Anmol Asrani
HADOOP-19354 S3A: InputStreams to be created by factory under S3AStore Major fs/s3 Steve Loughran Steve Loughran
HADOOP-19467 HADOOP-19467: [ABFS][FnsOverBlob] Fixing Config Name in Documenatation Major fs/azure Anuj Modi Anuj Modi
HADOOP-19443 ABFS: [FnsOverBlob][Tests] Update Test Scripts to Run Tests with Blob Endpoint Major fs/azure Anuj Modi Anuj Modi
HADOOP-19445 ABFS: [FnsOverBlob][Tests] Add Tests For Negative Scenarios Identified for Rename Operation Major fs/azure Anuj Modi Manish Bhatt
HADOOP-19185 Improve ABFS metric integration with iOStatistics Major fs/azure Steve Loughran Manish Bhatt
HADOOP-19348 S3A: Add initial support for analytics-accelerator-s3 Major fs/s3 Ahmar Suhail Ahmar Suhail
HADOOP-19303 VectorIO API to support releasing buffers on failure Major fs, fs/s3 Steve Loughran Steve Loughran
HADOOP-19446 ABFS: [FnsOverBlob][Tests] Add Tests For Negative Scenarios Identified for Delete Operation Major fs/azure Anuj Modi Manika Joshi
HADOOP-15224 S3A: Add option to set checksum on S3 objects Minor fs/s3 Steve Loughran Raphael Azzolini
HADOOP-19455 HADOOP-19455. S3A: Enable logging of SDK client metrics Minor fs/s3 Steve Loughran Steve Loughran
HADOOP-19512 S3A: Test failures during AWS SDK upgrade Minor fs/s3, test Steve Loughran Steve Loughran
HADOOP-19494 ABFS: Fix Case Sensitivity Issue for hdi_isfolder metadata Major fs/azure Manish Bhatt Manish Bhatt
HADOOP-19531 ABFS: [FnsOverBlob] Streaming List Path Result Should Happen Inside Retry Loop Major fs/azure Anuj Modi Anuj Modi
HADOOP-19474 ABFS: [FnsOverBlob] Listing Optimizations to avoid multiple iteration over list response. Major fs/azure Anuj Modi Anuj Modi
HADOOP-19444 ABFS: [FnsOverBlob][Tests] Add Tests For Negative Scenarios Identified for Ingress Operations Major fs/azure Anuj Modi Anmol Asrani
HADOOP-19515 ABFS: [FnsOverBlob] Updating Documentations of Hadoop Drivers for Azure Minor fs/azure Manika Joshi Manika Joshi
HADOOP-19256 S3A: Support S3 Conditional Writes Major fs/s3 Ahmar Suhail Saikat Roy
HADOOP-19543 ABFS: [FnsOverBlob] Remove Duplicates from Blob Endpoint Listing Across Iterations Blocker fs/azure Anuj Modi Anuj Modi
HADOOP-19522 ABFS: [FnsOverBlob] Rename Recovery Should Succeed When Marker File Exists with Destination Directory Blocker fs/azure Manish Bhatt Manish Bhatt
HADOOP-19518 ABFS: [FnsOverBlob] WASB to ABFS Migration Config Support Script Major fs/azure Manika Joshi Manika Joshi
HADOOP-19497 [ABFS] Enable rename and create recovery from client transaction id over DFS endpoint Major fs/azure Manish Bhatt Manish Bhatt
HADOOP-19471 ABFS: [FnsOverBlob] Support Fixed SAS token at container level Major fs/azure Manika Joshi Manika Joshi
HADOOP-19234 ABFS: [FnsOverBlob] Adding Integration Tests for Special Scenarios in Blob Endpoint Major fs/azure Anuj Modi Anuj Modi
HADOOP-19527 S3A: testVectoredReadAfterNormalRead() failing with 412 response from S3 Blocker fs/s3 Steve Loughran Ahmar Suhail
HADOOP-19485 S3A: Upgrade AWS V2 SDK to 2.29.52 Major build, fs/s3 Steve Loughran Steve Loughran
HADOOP-19406 ABFS: [FNS Over Blob] Support User Delegation SAS for FNS Blob Major fs/azure Anmol Asrani Manika Joshi
HADOOP-19557 S3A: S3ABlockOutputStream to never log/reject hflush(): calls Critical fs/s3 Steve Loughran Steve Loughran
HADOOP-19480 S3A Analytics-Accelerator: Upgrade AAL to 1.0.0 Major fs/s3 Ahmar Suhail
HADOOP-19542 S3A Analytics-Accelerator: AAL stream factory not being closed Blocker fs/s3 Ahmar Suhail Ahmar Suhail
HADOOP-19567 S3A: error stack traces printed on analytics stream factory close Major fs/s3 Steve Loughran Steve Loughran
HADOOP-19572 ABFS: [FnsOverBlob] Empty Page Issue on Subsequent ListBlob call Major fs/azure Anuj Modi Anuj Modi
HADOOP-19548 [ABFS] Fix logging in FSDataInputStream buffersize as that is not used and confusing the customer Major fs/azure, hadoop-common Mukund Thakur Manika Joshi
HADOOP-19580 ABFS: [FnsOverBlob][BugFix] IsNonEmptyDirectory Check should loop on listing using updated continuation token Critical fs/azure Anuj Modi Anuj Modi
HADOOP-18940 ABFS: Remove commons IOUtils.close() from AbfsOutputStream Critical fs/azure Steve Loughran Mehakmeet Singh
HADOOP-19295 S3A: fs.s3a.connection.request.timeout too low for large uploads over slow links Major fs/s3 Steve Loughran Steve Loughran
HADOOP-19575 ABFS: [FNSOverBlob] Add Distinct String In User Agent to Get Telemetry for FNS-Blob Major fs/azure Manika Joshi Manika Joshi
HADOOP-19627 S3A: testIfMatchOverwriteWithOutdatedEtag() fails when not using SSE-KMS Major fs/s3 Ahmar Suhail Ahmar Suhail
HADOOP-18296 Memory fragmentation in ChecksumFileSystem Vectored IO implementation. Minor common Mukund Thakur Steve Loughran
HADOOP-19626 S3A: AAL - Update to version 1.2.1 Major fs/s3 Ahmar Suhail Ahmar Suhail

OTHER:

JIRA Summary Priority Component Reporter Contributor
HADOOP-18568 Magic Committer optional clean up Minor fs/s3 André F. Sayed Mohammad Hossein Torabi
HADOOP-19123 Update commons-configuration2 to 2.10.1 due to CVE Major build PJ Fanning PJ Fanning
HADOOP-19315 Bump avro from 1.9.2 to 1.11.4 Major build Dominik Diedrich Dominik Diedrich
HADOOP-19335 Bump netty to 4.1.116 due to CVE-2024-47535 Major . PJ Fanning PJ Fanning
HADOOP-19380 Update the year to 2025 Major . Ayush Saxena Ayush Saxena
HDFS-17080 Fix ec connection leak (GitHub PR#5807) Major . Wei-Chiu Chuang Wei-Chiu Chuang
HADOOP-19456 Upgrade kafka to 3.9.0 to fix CVE-2024-31141 Major build Palakur Eshwitha Sai Palakur Eshwitha Sai
HADOOP-19465 upgrade to netty 4.1.118 due to CVE-2025-24970 Major build PJ Fanning PJ Fanning
HADOOP-19311 [ABFS] Implement Backoff and Read Footer metrics using IOStatistics Class Major fs/azure Manish Bhatt Manish Bhatt
HADOOP-19383 upgrade Mina 2.0.27 due to CVE-2024-52046 Major . PJ Fanning PJ Fanning
HADOOP-19450 [ABFS] Rename/Create path idempotency client-level resolution Major fs/azure Manish Bhatt Manish Bhatt
HADOOP-19634 acknowledge Guava license on LimitInputStream Major . PJ Fanning PJ Fanning