Apache Hadoop Changelog

Release 3.3.5 - 2023-03-14

IMPORTANT ISSUES:

JIRA Summary Priority Component Reporter Contributor
HADOOP-17956 Replace all default Charset usage with UTF-8 Major . Viraj Jasani Viraj Jasani
HADOOP-18621 CryptoOutputStream::close leak when encrypted zones + quota exceptions Critical fs Colm Dougan Colm Dougan

NEW FEATURES:

JIRA Summary Priority Component Reporter Contributor
HADOOP-18003 Add a method appendIfAbsent for CallerContext Minor . Tao Li Tao Li
HDFS-16331 Make dfs.blockreport.intervalMsec reconfigurable Major . Tao Li Tao Li
HDFS-16371 Exclude slow disks when choosing volume Major . Tao Li Tao Li
HDFS-16400 Reconfig DataXceiver parameters for datanode Major . Tao Li Tao Li
HDFS-16399 Reconfig cache report parameters for datanode Major . Tao Li Tao Li
HDFS-16398 Reconfig block report parameters for datanode Major . Tao Li Tao Li
HDFS-16396 Reconfig slow peer parameters for datanode Major . Tao Li Tao Li
HDFS-16397 Reconfig slow disk parameters for datanode Major . Tao Li Tao Li
MAPREDUCE-7341 Add a task-manifest output committer for Azure and GCS Major client Steve Loughran Steve Loughran
HADOOP-18163 hadoop-azure support for the Manifest Committer of MAPREDUCE-7341 Major fs/azure Steve Loughran Steve Loughran
HDFS-16413 Reconfig dfs usage parameters for datanode Major . Tao Li Tao Li
HDFS-16521 DFS API to retrieve slow datanodes Major . Viraj Jasani Viraj Jasani
HDFS-16568 dfsadmin -reconfig option to start/query reconfig on all live datanodes Major . Viraj Jasani Viraj Jasani
HDFS-16582 Expose aggregate latency of slow node as perceived by the reporting node Major . Viraj Jasani Viraj Jasani
HDFS-16595 Slow peer metrics - add median, mad and upper latency limits Major . Viraj Jasani Viraj Jasani
YARN-11241 Add uncleaning option for local app log file with log-aggregation enabled Major log-aggregation Ashutosh Gupta Ashutosh Gupta
HADOOP-18103 High performance vectored read API in Hadoop Major common, fs, fs/adl, fs/s3 Mukund Thakur Mukund Thakur

IMPROVEMENTS:

JIRA Summary Priority Component Reporter Contributor
HADOOP-17276 Extend CallerContext to make it include many items Major . Hui Fei Hui Fei
HDFS-15745 Make DataNodePeerMetrics#LOW_THRESHOLD_MS and MIN_OUTLIER_DETECTION_NODES configurable Major . Haibin Huang Haibin Huang
HDFS-16266 Add remote port information to HDFS audit log Major . Tao Li Tao Li
YARN-10997 Revisit allocation and reservation logging Major . Andras Gyori Andras Gyori
HDFS-16310 RBF: Add client port to CallerContext for Router Major . Tao Li Tao Li
HDFS-16352 return the real datanode numBlocks in #getDatanodeStorageReport Major . qinyuren qinyuren
HDFS-16426 fix nextBlockReportTime when trigger full block report force Major . qinyuren qinyuren
HDFS-16430 Validate maximum blocks in EC group when adding an EC policy Minor ec, erasure-coding daimin daimin
HDFS-16403 Improve FUSE IO performance by supporting FUSE parameter max_background Minor fuse-dfs daimin daimin
HDFS-16262 Async refresh of cached locations in DFSInputStream Major . Bryan Beaudreault Bryan Beaudreault
HADOOP-18093 Better exception handling for testFileStatusOnMountLink() in ViewFsBaseTest.java Trivial . Xing Lin Xing Lin
HDFS-16423 balancer should not get blocks on stale storages Major balancer & mover qinyuren qinyuren
HADOOP-18139 Allow configuration of zookeeper server principal Major auth Owen O’Malley Owen O’Malley
YARN-11076 Upgrade jQuery version in Yarn UI2 Major yarn-ui-v2 Tamas Domok Tamas Domok
HDFS-16495 RBF should prepend the client ip rather than append it. Major . Owen O’Malley Owen O’Malley
HADOOP-18144 getTrashRoot/s in ViewFileSystem should return viewFS path, not targetFS path Major common Xing Lin Xing Lin
HADOOP-18162 hadoop-common enhancements for the Manifest Committer of MAPREDUCE-7341 Major fs Steve Loughran Steve Loughran
HDFS-16529 Remove unnecessary setObserverRead in TestConsistentReadsObserver Trivial test Zhaohui Wang Zhaohui Wang
HDFS-16530 setReplication debug log creates a new string even if debug is disabled Major namenode Stephen O’Donnell Stephen O’Donnell
HDFS-16457 Make fs.getspaceused.classname reconfigurable Major namenode yanbin.zhang yanbin.zhang
HDFS-16427 Add debug log for BlockManager#chooseExcessRedundancyStriped Minor erasure-coding Tao Li Tao Li
HDFS-16497 EC: Add param comment for liveBusyBlockIndices with HDFS-14768 Minor erasure-coding, namanode caozhiqiang caozhiqiang
HDFS-16389 Improve NNThroughputBenchmark test mkdirs Major benchmarks, namenode JiangHua Zhu JiangHua Zhu
HADOOP-17551 Upgrade maven-site-plugin to 3.11.0 Major . Akira Ajisaka Ashutosh Gupta
HDFS-16519 Add throttler to EC reconstruction Minor datanode, ec daimin daimin
HDFS-14478 Add libhdfs APIs for openFile Major hdfs-client, libhdfs, native Sahil Takiar Sahil Takiar
HADOOP-16202 Enhance openFile() for better read performance against object stores Major fs, fs/s3, tools/distcp Steve Loughran Steve Loughran
YARN-11116 Migrate Times util from SimpleDateFormat to thread-safe DateTimeFormatter class Minor . Jonathan Turner Eagles Jonathan Turner Eagles
HDFS-16520 Improve EC pread: avoid potential reading whole block Major dfsclient, ec, erasure-coding daimin daimin
HADOOP-18167 Add metrics to track delegation token secret manager operations Major . Hector Sandoval Chaverri Hector Sandoval Chaverri
YARN-10080 Support show app id on localizer thread pool Major nodemanager zhoukang Ashutosh Gupta
HADOOP-18172 Change scope of getRootFallbackLink for InodeTree to make them accessible from outside package Minor . Xing Lin Xing Lin
HDFS-16588 Backport HDFS-16584 to branch-3.3. Major balancer & mover, namenode JiangHua Zhu JiangHua Zhu
HADOOP-18240 Upgrade Yetus to 0.14.0 Major build Akira Ajisaka Ashutosh Gupta
HDFS-16585 Add @VisibleForTesting in Dispatcher.java after HDFS-16268 Trivial . Wei-Chiu Chuang Ashutosh Gupta
HADOOP-18244 Fix Hadoop-Common JavaDoc Error on branch-3.3 Major common Shilun Fan Shilun Fan
HADOOP-18269 Misleading method name in DistCpOptions Minor tools/distcp guophilipse guophilipse
HADOOP-18275 update os-maven-plugin to 1.7.0 Minor build Steve Loughran Steve Loughran
HDFS-16610 Make fsck read timeout configurable Major hdfs-client Stephen O’Donnell Stephen O’Donnell
HDFS-16576 Remove unused imports in HDFS project Minor . Ashutosh Gupta Ashutosh Gupta
HDFS-16629 [JDK 11] Fix javadoc warnings in hadoop-hdfs module Minor hdfs Shilun Fan Shilun Fan
YARN-11172 Fix testDelegationToken Major test zhengchenyu zhengchenyu
HADOOP-17833 Improve Magic Committer Performance Minor fs/s3 Steve Loughran Steve Loughran
HADOOP-18288 Total requests and total requests per sec served by RPC servers Major . Viraj Jasani Viraj Jasani
HADOOP-18336 tag FSDataInputStream.getWrappedStream() @Public/@Stable Minor fs Steve Loughran Ashutosh Gupta
HADOOP-13144 Enhancing IPC client throughput via multiple connections per user Minor ipc Jason Kace Íñigo Goiri
HDFS-16712 Fix incorrect placeholder in DataNode.java Major . ZanderXu ZanderXu
HDFS-16702 MiniDFSCluster should report cause of exception in assertion error Minor hdfs Steve Vaughan Steve Vaughan
HADOOP-18365 Updated addresses are still accessed using the old IP address Major common Steve Vaughan Steve Vaughan
HDFS-16687 RouterFsckServlet replicates code from DfsServlet base class Major federation Steve Vaughan Steve Vaughan
HADOOP-18333 hadoop-client-runtime impact by CVE-2022-2047 CVE-2022-2048 due to shaded jetty Major build phoebe chen Ashutosh Gupta
HADOOP-18406 Adds alignment context to call path for creating RPC proxy with multiple connections per user. Major ipc Simbarashe Dzinamarira Simbarashe Dzinamarira
HDFS-16684 Exclude self from JournalNodeSyncer when using a bind host Major journal-node Steve Vaughan Steve Vaughan
HDFS-16686 GetJournalEditServlet fails to authorize valid Kerberos request Major journal-node Steve Vaughan Steve Vaughan
YARN-11303 Upgrade jquery ui to 1.13.2 Major security D M Murali Krishna Reddy Ashutosh Gupta
HADOOP-16769 LocalDirAllocator to provide diagnostics when file creation fails Minor util Ramesh Kumar Thangarajan Ashutosh Gupta
HADOOP-18341 upgrade commons-configuration2 to 2.8.0 and commons-text to 1.9 Major . PJ Fanning PJ Fanning
HDFS-16776 Erasure Coding: The length of targets should be checked when DN gets a reconstruction task Major . Kidd5368 Kidd5368
HADOOP-18469 Add XMLUtils methods to centralise code that creates secure XML parsers Major . PJ Fanning PJ Fanning
HADOOP-18442 Remove the hadoop-openstack module Major build, fs, fs/swift Steve Loughran Steve Loughran
HADOOP-18468 upgrade jettison json jar due to fix CVE-2022-40149 Major build PJ Fanning PJ Fanning
HADOOP-17779 Lock File System Creator Semaphore Uninterruptibly Minor fs David Mollitor David Mollitor
HADOOP-18360 Update commons-csv from 1.0 to 1.9.0. Minor common Shilun Fan Shilun Fan
HADOOP-18493 update jackson-databind 2.12.7.1 due to CVE fixes Major . PJ Fanning PJ Fanning
HADOOP-17563 Update Bouncy Castle to 1.68 or later Major build Takanobu Asanuma PJ Fanning
HADOOP-18497 Upgrade commons-text version to fix CVE-2022-42889 Major build Xiaoqiao He PJ Fanning
HDFS-16795 Use secure XML parser utils in hdfs classes Major . PJ Fanning PJ Fanning
YARN-11330 Use secure XML parser utils in YARN Major . PJ Fanning PJ Fanning
MAPREDUCE-7411 Use secure XML parser utils in MapReduce Major . PJ Fanning PJ Fanning
HADOOP-18512 upgrade woodstox-core to 5.4.0 for security fix Major common phoebe chen PJ Fanning
YARN-11363 Remove unused TimelineVersionWatcher and TimelineVersion from hadoop-yarn-server-tests Major test, yarn Ashutosh Gupta Ashutosh Gupta
YARN-11364 Docker Container to accept docker Image name with sha256 digest Major yarn Ashutosh Gupta Ashutosh Gupta
HADOOP-18517 ABFS: Add fs.azure.enable.readahead option to disable readahead Major fs/azure Steve Loughran Steve Loughran
HADOOP-18484 upgrade hsqldb to v2.7.1 due to CVE Major . PJ Fanning Ashutosh Gupta
HDFS-16844 [RBF] The routers should be resiliant against exceptions from StateStore Major rbf Owen O’Malley Owen O’Malley
HADOOP-18573 Improve error reporting on non-standard kerberos names Blocker security Steve Loughran Steve Loughran
HADOOP-18561 CVE-2021-37533 on commons-net is included in hadoop common and hadoop-client-runtime Blocker build phoebe chen Steve Loughran
HADOOP-18067 Über-jira: S3A Hadoop 3.3.5 features Major fs/s3 Steve Loughran Mukund Thakur
YARN-10444 Node Manager to use openFile() with whole-file read policy for localizing files. Minor nodemanager Steve Loughran Steve Loughran
HADOOP-18661 Fix bin/hadoop usage script terminology Blocker scripts Steve Loughran Steve Loughran

BUG FIXES:

JIRA Summary Priority Component Reporter Contributor
HADOOP-17116 Skip Retry INFO logging on first failover from a proxy Major ha Hanisha Koneru Hanisha Koneru
YARN-10553 Refactor TestDistributedShell Major distributed-shell, test Ahmed Hussein Ahmed Hussein
HDFS-15839 RBF: Cannot get method setBalancerBandwidth on Router Client Major rbf Yang Yun Yang Yun
HADOOP-17588 CryptoInputStream#close() should be synchronized Major . Renukaprasad C Renukaprasad C
HADOOP-17836 Improve logging on ABFS error reporting Minor fs/azure Steve Loughran Steve Loughran
HADOOP-17989 ITestAzureBlobFileSystemDelete failing “Operations has null HTTP response” Major fs/azure, test Steve Loughran Steve Loughran
YARN-11055 In cgroups-operations.c some fprintf format strings don’t end with “\n” Minor nodemanager Gera Shegalov Gera Shegalov
YARN-11065 Bump follow-redirects from 1.13.3 to 1.14.7 in hadoop-yarn-ui Major yarn-ui-v2 Akira Ajisaka
HDFS-16303 Losing over 100 datanodes in state decommissioning results in full blockage of all datanode decommissioning Major . Kevin Wikant Kevin Wikant
HDFS-16443 Fix edge case where DatanodeAdminDefaultMonitor doubly enqueues a DatanodeDescriptor on exception Major hdfs Kevin Wikant Kevin Wikant
HDFS-16449 Fix hadoop web site release notes and changelog not available Minor documentation guophilipse guophilipse
YARN-10788 TestCsiClient fails Major test Akira Ajisaka Akira Ajisaka
HADOOP-18126 Update junit 5 version due to build issues Major bulid PJ Fanning PJ Fanning
YARN-11033 isAbsoluteResource is not correct for dynamically created queues Minor yarn Tamas Domok Tamas Domok
YARN-10894 Follow up YARN-10237: fix the new test case in TestRMWebServicesCapacitySched Major . Tamas Domok Tamas Domok
YARN-11022 Fix the documentation for max-parallel-apps in CS Major capacity scheduler Tamas Domok Tamas Domok
HADOOP-18150 Fix ITestAuditManagerDisabled after S3A audit logging was enabled in HADOOP-18091 Major fs/s3 Mehakmeet Singh Mehakmeet Singh
HADOOP-17976 abfs etag extraction inconsistent between LIST and HEAD calls Minor fs/azure Steve Loughran Steve Loughran
HADOOP-18129 Change URI[] in INodeLink to String[] to reduce memory footprint of ViewFileSystem Major . Abhishek Das Abhishek Das
HADOOP-18145 Fileutil’s unzip method causes unzipped files to lose their original permissions Major common jingxiong zhong jingxiong zhong
HDFS-16518 KeyProviderCache close cached KeyProvider with Hadoop ShutdownHookManager Major hdfs Lei Yang Lei Yang
HADOOP-18169 getDelegationTokens in ViewFs should also fetch the token from the fallback FS Major . Xing Lin Xing Lin
HDFS-16479 EC: NameNode should not send a reconstruction work when the source datanodes are insufficient Critical ec, erasure-coding Yuanbo Liu Takanobu Asanuma
HDFS-16509 Fix decommission UnsupportedOperationException: Remove unsupported Major namenode daimin daimin
HDFS-16456 EC: Decommission a rack with only on dn will fail when the rack number is equal with replication Critical ec, namenode caozhiqiang caozhiqiang
HADOOP-18201 Remove base and bucket overrides for endpoint in ITestS3ARequesterPays.java Major fs/s3 Mehakmeet Singh Daniel Carl Jones
HDFS-16536 TestOfflineImageViewer fails on branch-3.3 Major test Akira Ajisaka Ashutosh Gupta
HDFS-16538 EC decoding failed due to not enough valid inputs Major erasure-coding qinyuren qinyuren
HDFS-16544 EC decoding failed due to invalid buffer Major erasure-coding qinyuren qinyuren
HADOOP-17564 Fix typo in UnixShellGuide.html Trivial . Takanobu Asanuma Ashutosh Gupta
HDFS-16552 Fix NPE for TestBlockManager Major . Tao Li Tao Li
MAPREDUCE-7246 In MapredAppMasterRest#Mapreduce_Application_Master_Info_API, the datatype of appId should be “string”. Major documentation jenny Ashutosh Gupta
YARN-10187 Removing hadoop-yarn-project/hadoop-yarn/README as it is no longer maintained. Minor documentation N Sanketh Reddy Ashutosh Gupta
HADOOP-16515 Update the link to compatibility guide Minor documentation Akira Ajisaka Ashutosh Gupta
HDFS-16185 Fix comment in LowRedundancyBlocks.java Minor documentation Akira Ajisaka Ashutosh Gupta
HADOOP-17479 Fix the examples of hadoop config prefix Minor documentation Akira Ajisaka Ashutosh Gupta
HADOOP-18222 Prevent DelegationTokenSecretManagerMetrics from registering multiple times Major . Hector Sandoval Chaverri Hector Sandoval Chaverri
HDFS-16540 Data locality is lost when DataNode pod restarts in kubernetes Major namenode Huaxiang Sun Huaxiang Sun
YARN-11133 YarnClient gets the wrong EffectiveMinCapacity value Major api Zilong Zhu Zilong Zhu
YARN-10850 TimelineService v2 lists containers for all attempts when filtering for one Major timelinereader Benjamin Teke Benjamin Teke
YARN-11141 Capacity Scheduler does not support ambiguous queue names when moving application across queues Major capacity scheduler András Győri András Győri
HDFS-16586 Purge FsDatasetAsyncDiskService threadgroup; it causes BPServiceActor$CommandProcessingThread IllegalThreadStateException ‘fatal exception and exit’ Major datanode Michael Stack Michael Stack
HADOOP-18251 Fix failure of extracting JIRA id from commit message in git_jira_fix_version_check.py Minor build Masatake Iwasaki Masatake Iwasaki
YARN-11128 Fix comments in TestProportionalCapacityPreemptionPolicy* Minor capacityscheduler, documentation Ashutosh Gupta Ashutosh Gupta
HADOOP-18234 s3a access point xml examples are wrong Minor documentation, fs/s3 Steve Loughran Ashutosh Gupta
HADOOP-18238 Fix reentrancy check in SFTPFileSystem.close() Major common yi liu Ashutosh Gupta
HDFS-16583 DatanodeAdminDefaultMonitor can get stuck in an infinite loop Major . Stephen O’Donnell Stephen O’Donnell
HDFS-16608 Fix the link in TestClientProtocolForPipelineRecovery Minor documentation Samrat Deb Samrat Deb
HDFS-16563 Namenode WebUI prints sensitive information on Token Expiry Major namanode, security, webhdfs Renukaprasad C Renukaprasad C
HDFS-16623 IllegalArgumentException in LifelineSender Major . ZanderXu ZanderXu
HDFS-16064 Determine when to invalidate corrupt replicas based on number of usable replicas Major datanode, namenode Kevin Wikant Kevin Wikant
HADOOP-18255 fsdatainputstreambuilder.md refers to hadoop 3.3.3, when it shouldn’t Minor documentation Steve Loughran Ashutosh Gupta
MAPREDUCE-7387 Fix TestJHSSecurity#testDelegationToken AssertionError due to HDFS-16563 Major . Shilun Fan Shilun Fan
MAPREDUCE-7369 MapReduce tasks timing out when spends more time on MultipleOutputs#close Major . Prabhu Joseph Ashutosh Gupta
MAPREDUCE-7391 TestLocalDistributedCacheManager failing after HADOOP-16202 Major test Steve Loughran Steve Loughran
HDFS-16591 StateStoreZooKeeper fails to initialize Major rbf Hector Sandoval Chaverri Hector Sandoval Chaverri
HADOOP-18321 Fix when to read an additional record from a BZip2 text file split Critical io Ashutosh Gupta Ashutosh Gupta
HADOOP-18100 Change scope of inner classes in InodeTree to make them accessible outside package Major . Abhishek Das Abhishek Das
HADOOP-18217 shutdownhookmanager should not be multithreaded (deadlock possible) Minor util Catherinot Remi
MAPREDUCE-7372 MapReduce set permission too late in copyJar method Major mrv2 Zhang Dongsheng
HADOOP-18330 S3AFileSystem removes Path when calling createS3Client Minor fs/s3 Ashutosh Pant Ashutosh Pant
HADOOP-18390 Fix out of sync import for HADOOP-18321 Minor . Ashutosh Gupta Ashutosh Gupta
HADOOP-18340 deleteOnExit does not work with S3AFileSystem Minor fs/s3 Huaxiang Sun Huaxiang Sun
HADOOP-18383 Codecs with @DoNotPool annotation are not closed causing memory leak Major common Kevin Sewell Kevin Sewell
HDFS-16729 RBF: fix some unreasonably annotated docs Major documentation, rbf JiangHua Zhu JiangHua Zhu
HADOOP-18398 Prevent AvroRecord*.class from being included non-test jar Major common YUBI LEE YUBI LEE
HDFS-4043 Namenode Kerberos Login does not use proper hostname for host qualified hdfs principal name. Major security Ahad Rana Steve Vaughan
MAPREDUCE-7403 Support spark dynamic partitioning in the Manifest Committer Major mrv2 Steve Loughran Steve Loughran
HDFS-16732 [SBN READ] Avoid get location from observer when the block report is delayed. Critical hdfs zhengchenyu zhengchenyu
HADOOP-18375 Fix failure of shelltest for hadoop_add_ldlibpath Minor test Masatake Iwasaki Masatake Iwasaki
HDFS-16755 TestQJMWithFaults.testUnresolvableHostName() can fail due to unexpected host resolution Minor test Steve Vaughan Steve Vaughan
HADOOP-18400 Fix file split duplicating records from a succeeding split when reading BZip2 text files Critical . Ashutosh Gupta Ashutosh Gupta
HADOOP-18242 ABFS Rename Failure when tracking metadata is in incomplete state Major fs/azure Mehakmeet Singh Mehakmeet Singh
HADOOP-18456 NullPointerException in ObjectListingIterator’s constructor Blocker fs/s3 Quanlong Huang Steve Loughran
HADOOP-18444 Add Support for localized trash for ViewFileSystem in Trash.moveToAppropriateTrash Major . Xing Lin Xing Lin
HADOOP-18443 Upgrade snakeyaml to 1.32 Major security Ashutosh Gupta Ashutosh Gupta
HDFS-16766 hdfs ec command loads (administrator provided) erasure code policy files without disabling xml entity expansion Major security Jing Ashutosh Gupta
HDFS-13369 FSCK Report broken with RequestHedgingProxyProvider Major hdfs Harshakiran Reddy Ranith Sardar
YARN-11039 LogAggregationFileControllerFactory::getFileControllerForRead can leak threads Blocker log-aggregation Rajesh Balamohan Steve Loughran
HADOOP-18499 S3A to support HTTPS web proxies Major fs/s3 Mehakmeet Singh Mehakmeet Singh
HADOOP-18233 Possible race condition with TemporaryAWSCredentialsProvider Major auth, fs/s3 Jason Sleight Jimmy Wong
MAPREDUCE-7425 Document Fix for yarn.app.mapreduce.client-am.ipc.max-retries Major yarn teng wang teng wang
HADOOP-18528 Disable abfs prefetching by default Major fs/azure Mehakmeet Singh Mehakmeet Singh
HDFS-16836 StandbyCheckpointer can still trigger rollback fs image after RU is finalized Major hdfs Lei Yang Lei Yang
HADOOP-18324 Interrupting RPC Client calls can lead to thread exhaustion Critical ipc Owen O’Malley Owen O’Malley
HDFS-16832 [SBN READ] Fix NPE when check the block location of empty directory Major . zhengchenyu zhengchenyu
HADOOP-18498 [ABFS]: Error introduced when SAS Token containing ‘?’ prefix is passed Minor fs/azure Sree Bhattacharyya Sree Bhattacharyya
HDFS-16847 RBF: StateStore writer should not commit tmp fail if there was an error in writing the file. Critical hdfs, rbf Simbarashe Dzinamarira Simbarashe Dzinamarira
HADOOP-18401 No ARM binaries in branch-3.3.x releases Minor build Ling Xu
HADOOP-18408 [ABFS]: ITestAbfsManifestCommitProtocol fails on nonHNS configuration Minor fs/azure, test Pranav Saxena Sree Bhattacharyya
HADOOP-18402 S3A committer NPE in spark job abort Blocker fs/s3 Steve Loughran Steve Loughran
HADOOP-18569 NFS Gateway may release buffer too early Blocker nfs Attila Doroszlai Attila Doroszlai
HADOOP-18574 Changing log level of IOStatistics increment to make the DEBUG logs less noisy Major fs/s3 Mehakmeet Singh Mehakmeet Singh
HADOOP-18521 ABFS ReadBufferManager buffer sharing across concurrent HTTP requests Critical fs/azure Steve Loughran Steve Loughran
MAPREDUCE-7375 JobSubmissionFiles don’t set right permission after mkdirs Major mrv2 Zhang Dongsheng
HADOOP-17717 Update wildfly openssl to 1.1.3.Final Major . Wei-Chiu Chuang Wei-Chiu Chuang
HADOOP-18598 maven site generation doesn’t include javadocs Blocker site Steve Loughran Steve Loughran
HDFS-16895 NamenodeHeartbeatService should use credentials of logged in user Major rbf Hector Sandoval Chaverri Hector Sandoval Chaverri
HDFS-16853 The UT TestLeaseRecovery2#testHardLeaseRecoveryAfterNameNodeRestart failed because HADOOP-18324 Blocker . ZanderXu ZanderXu
HADOOP-18641 cyclonedx maven plugin breaks builds on recent maven releases (3.9.0) Major build Steve Loughran Steve Loughran
HDFS-16923 The getListing RPC will throw NPE if the path does not exist Critical . ZanderXu ZanderXu
HDFS-16896 HDFS Client hedged read has increased failure rate than without hedged read Major hdfs-client Tom McCormick Tom McCormick
YARN-11383 Workflow priority mappings is case sensitive Major yarn Aparajita Choudhary Aparajita Choudhary
HDFS-16939 Fix the thread safety bug in LowRedundancyBlocks Major namanode Shuyan Zhang Shuyan Zhang
HDFS-16934 org.apache.hadoop.hdfs.tools.TestDFSAdmin#testAllDatanodesReconfig regression Minor dfsadmin, test Steve Loughran Shilun Fan

TESTS:

JIRA Summary Priority Component Reporter Contributor
HDFS-16573 Fix test TestDFSStripedInputStreamWithRandomECPolicy Minor test daimin daimin
HDFS-16637 TestHDFSCLI#testAll consistently failing Major . Viraj Jasani Viraj Jasani
YARN-11248 Add unit test for FINISHED_CONTAINERS_PULLED_BY_AM event on DECOMMISSIONING Major test Ashutosh Gupta Ashutosh Gupta
HDFS-16625 Unit tests aren’t checking for PMDK availability Major test Steve Vaughan Steve Vaughan

SUB-TASKS:

JIRA Summary Priority Component Reporter Contributor
HDFS-13293 RBF: The RouterRPCServer should transfer client IP via CallerContext to NamenodeRpcServer Major rbf Baolong Mao Hui Fei
HDFS-15630 RBF: Fix wrong client IP info in CallerContext when requests mount points with multi-destinations. Major rbf Chengwei Wang Chengwei Wang
HADOOP-17152 Implement wrapper for guava newArrayList and newLinkedList Major common Ahmed Hussein Viraj Jasani
HADOOP-17851 S3A to support user-specified content encoding Minor fs/s3 Holden Karau Holden Karau
HADOOP-17492 abfs listLocatedStatus to support incremental/async page fetching Major fs/azure Steve Loughran Steve Loughran
HADOOP-17409 Remove S3Guard - no longer needed Major fs/s3 Steve Loughran Steve Loughran
HADOOP-18084 ABFS: Add testfilePath while verifying test contents are read correctly Minor fs/azure, test Anmol Asrani Anmol Asrani
HDFS-16169 Fix TestBlockTokenWithDFSStriped#testEnd2End failure Major test Hui Fei secfree
HADOOP-18091 S3A auditing leaks memory through ThreadLocal references Major fs/s3 Steve Loughran Steve Loughran
HADOOP-18071 ABFS: Set driver global timeout for ITestAzureBlobFileSystemBasics Major fs/azure Sumangala Patki Sumangala Patki
HADOOP-17765 ABFS: Use Unique File Paths in Tests Major fs/azure Sumangala Patki Sumangala Patki
HADOOP-17862 ABFS: Fix unchecked cast compiler warning for AbfsListStatusRemoteIterator Major fs/azure Sumangala Patki Sumangala Patki
HADOOP-18075 ABFS: Fix failure caused by listFiles() in ITestAbfsRestOperationException Major fs/azure Sumangala Patki Sumangala Patki
HADOOP-18112 Implement paging during S3 multi object delete. Critical fs/s3 Mukund Thakur Mukund Thakur
HADOOP-16204 ABFS tests to include terasort Minor fs/azure, test Steve Loughran Steve Loughran
HDFS-13248 RBF: Namenode need to choose block location for the client Major . Wu Weiwei Owen O’Malley
HADOOP-13704 S3A getContentSummary() to move to listFiles(recursive) to count children; instrument use Minor fs/s3 Steve Loughran Ahmar Suhail
HADOOP-14661 S3A to support Requester Pays Buckets Minor common, util Mandus Momberg Daniel Carl Jones
HDFS-16484 [SPS]: Fix an infinite loop bug in SPSPathIdProcessor thread Major . qinyuren qinyuren
HADOOP-17682 ABFS: Support FileStatus input to OpenFileWithOptions() via OpenFileParameters Major fs/azure Sumangala Patki Sumangala Patki
HADOOP-15983 Use jersey-json that is built to use jackson2 Major build Akira Ajisaka PJ Fanning
HADOOP-18104 Add configs to configure minSeekForVectorReads and maxReadSizeForVectorReads Major common, fs Mukund Thakur Mukund Thakur
HADOOP-18168 ITestMarkerTool.testRunLimitedLandsatAudit failing due to most of bucket content purged Minor fs/s3, test Steve Loughran Daniel Carl Jones
HADOOP-12020 Support configuration of different S3 storage classes Major fs/s3 Yann Landrin-Schweitzer Monthon Klongklaew
HADOOP-18105 Implement a variant of ElasticByteBufferPool which uses weak references for garbage collection. Major common, fs Mukund Thakur Mukund Thakur
HADOOP-18107 Vectored IO support for large S3 files. Major fs/s3 Mukund Thakur Mukund Thakur
HADOOP-18106 Handle memory fragmentation in S3 Vectored IO implementation. Major fs/s3 Mukund Thakur Mukund Thakur
HADOOP-17461 Add thread-level IOStatistics Context Major fs, fs/azure, fs/s3 Steve Loughran Mehakmeet Singh
HADOOP-18372 ILoadTestS3ABulkDeleteThrottling failing Minor fs/s3, test Steve Loughran Ahmar Suhail
HADOOP-18368 ITestCustomSigner fails when access point name has ‘-’ Minor . Ahmar Suhail Ahmar Suhail
HADOOP-15964 Add S3A support for Async Scatter/Gather IO Major fs/s3 Steve Loughran Mukund Thakur
HADOOP-18366 ITestS3Select.testSelectSeekFullLandsat is timing out Minor . Ahmar Suhail Ahmar Suhail
HADOOP-18373 IOStatisticsContext tuning Minor fs/s3, test Steve Loughran Viraj Jasani
HADOOP-18227 Add input stream IOstats for vectored IO api in S3A. Major fs/s3 Mukund Thakur Mukund Thakur
HADOOP-18392 Propagate vectored s3a input stream stats to file system stats. Major fs/s3 Mukund Thakur Mukund Thakur
HADOOP-18355 Update previous index properly while validating overlapping ranges. Major common, fs/s3 Mukund Thakur Mukund Thakur
HADOOP-18371 s3a FS init logs at warn if fs.s3a.create.storage.class is unset Blocker fs/s3 Steve Loughran Viraj Jasani
HADOOP-18385 ITestS3ACannedACLs failure; not in a span Major fs/s3, test Steve Loughran Ashutosh Gupta
HADOOP-18403 Fix FileSystem leak in ITestS3AAWSCredentialsProvider Minor fs/s3 Viraj Jasani Viraj Jasani
HADOOP-17882 distcp to use openFile() with sequential IO; ranges of reads Major tools/distcp Steve Loughran Steve Loughran
HADOOP-18391 Improve VectoredReadUtils#readVectored() for direct buffers Major fs Steve Loughran Mukund Thakur
HADOOP-18407 Improve vectored IO api spec. Minor fs, fs/s3 Mukund Thakur Mukund Thakur
HADOOP-18339 S3A storage class option only picked up when buffering writes to disk Major fs/s3 Steve Loughran Monthon Klongklaew
HADOOP-18410 S3AInputStream.unbuffer() async drain not releasing http connections Blocker fs/s3 Steve Loughran Steve Loughran
HADOOP-18439 Fix VectoredIO for LocalFileSystem when checksum is enabled. Major common Mukund Thakur Mukund Thakur
HADOOP-18416 ITestS3AIOStatisticsContext failure Major fs/s3, test Steve Loughran Mehakmeet Singh
HADOOP-18347 Restrict vectoredIO threadpool to reduce memory pressure Major common, fs, fs/adl, fs/s3 Rajesh Balamohan Mukund Thakur
HADOOP-18463 Add an integration test to process data asynchronously during vectored read. Major . Mukund Thakur Mukund Thakur
HADOOP-15460 S3A FS to add “fs.s3a.create.performance” to the builder file creation option set Major fs/s3 Steve Loughran Steve Loughran
HADOOP-18382 Upgrade AWS SDK to V2 - Prerequisites Minor . Ahmar Suhail Ahmar Suhail
HADOOP-18480 upgrade AWS SDK to 1.12.316 Major build, fs/s3 Steve Loughran Steve Loughran
HADOOP-18460 ITestS3AContractVectoredRead.testStopVectoredIoOperationsUnbuffer failing Minor fs/s3, test Steve Loughran Mukund Thakur
HADOOP-18488 Cherrypick HADOOP-11245 to branch-3.3 Major . Wei-Chiu Chuang Ashutosh Gupta
HADOOP-18481 AWS v2 SDK upgrade log to not warn of use standard AWS Credential Providers Major fs/s3 Steve Loughran Ahmar Suhail
HADOOP-18476 Abfs and S3A FileContext bindings to close wrapped filesystems in finalizer Blocker fs/azure, fs/s3 Steve Loughran Steve Loughran
HADOOP-18304 Improve S3A committers documentation clarity Trivial documentation Daniel Carl Jones Daniel Carl Jones
HADOOP-18465 S3A server-side encryption tests fail before checking encryption tests should skip Minor fs/s3, test Daniel Carl Jones Daniel Carl Jones
HADOOP-18530 ChecksumFileSystem::readVectored might return byte buffers not positioned at 0 Blocker fs Harshit Gupta Harshit Gupta
HADOOP-18457 ABFS: Support for account level throttling Major . Anmol Asrani Anmol Asrani
HADOOP-18560 AvroFSInput opens a stream twice and discards the second one without closing Blocker fs Steve Loughran Steve Loughran
HADOOP-18526 Leak of S3AInstrumentation instances via hadoop Metrics references Blocker fs/s3 Steve Loughran Steve Loughran
HADOOP-18546 disable purging list of in progress reads in abfs stream closed Blocker fs/azure Steve Loughran Pranav Saxena
HADOOP-18577 ABFS: add probes of readahead fix Major fs/azure Steve Loughran Steve Loughran
HADOOP-11867 Add a high-performance vectored read API. Major fs, fs/azure, fs/s3, hdfs-client Gopal Vijayaraghavan Mukund Thakur
HADOOP-18507 VectorIO FileRange type to support a “reference” field Major fs Steve Loughran Steve Loughran
HADOOP-18627 site intro docs to make clear Kerberos is mandatory for secure clusters Major site Steve Loughran Arnout Engelen
HADOOP-17584 s3a magic committer may commit more data Major fs/s3 yinan zhan Steve Loughran
HADOOP-18642 Cut excess dependencies from hadoop-azure, hadoop-aliyun transitive imports; fix LICENSE-binary Blocker build, fs/azure, fs/oss Steve Loughran Steve Loughran

OTHER:

JIRA Summary Priority Component Reporter Contributor
HDFS-15854 Make some parameters configurable for SlowDiskTracker and SlowPeerTracker Major . Tao Li Tao Li
YARN-10747 Bump YARN CSI protobuf version to 3.7.1 Major . Siyao Meng Siyao Meng
HDFS-16139 Update BPServiceActor Scheduler’s nextBlockReportTime atomically Major . Viraj Jasani Viraj Jasani
HADOOP-18014 CallerContext should not include some characters Major . Takanobu Asanuma Takanobu Asanuma
MAPREDUCE-7371 DistributedCache alternative APIs should not use DistributedCache APIs internally Major . Viraj Jasani Viraj Jasani
HADOOP-18114 Documentation Syntax Error Fix > AWS Assumed Roles Trivial documentation, fs/s3 Joey Krabacher Joey Krabacher
HDFS-16481 Provide support to set Http and Rpc ports in MiniJournalCluster Major . Viraj Jasani Viraj Jasani
HDFS-16502 Reconfigure Block Invalidate limit Major . Viraj Jasani Viraj Jasani
HDFS-16522 Set Http and Ipc ports for Datanodes in MiniDFSCluster Major . Viraj Jasani Viraj Jasani
HADOOP-18191 Log retry count while handling exceptions in RetryInvocationHandler Minor . Viraj Jasani Viraj Jasani
HDFS-16551 Backport HADOOP-17588 to 3.3 and other active old branches. Major . Renukaprasad C Renukaprasad C
HDFS-16618 sync_file_range error should include more volume and file info Minor . Viraj Jasani Viraj Jasani
HADOOP-18300 Update google-gson to 2.9.0 Minor build Igor Dvorzhak Igor Dvorzhak
HADOOP-18397 Shutdown AWSSecurityTokenService when its resources are no longer in use Major fs/s3 Viraj Jasani Viraj Jasani
HADOOP-18575 Make XML transformer factory more lenient Major common PJ Fanning PJ Fanning
HADOOP-18586 Update the year to 2023 Major . Ayush Saxena Ayush Saxena
HADOOP-18587 upgrade to jettison 1.5.3 to fix CVE-2022-40150 Major common PJ Fanning PJ Fanning