Apache Hadoop Changelog

Release 3.3.1 - 2021-06-13

IMPORTANT ISSUES:

JIRA Summary Priority Component Reporter Contributor
HADOOP-17338 Intermittent S3AInputStream failures: Premature end of Content-Length delimited message body etc Major fs/s3 Yongjun Zhang Yongjun Zhang
HDFS-15380 RBF: Could not fetch real remote IP in RouterWebHdfsMethods Major webhdfs tomscut tomscut

NEW FEATURES:

JIRA Summary Priority Component Reporter Contributor
HADOOP-16916 ABFS: Delegation SAS generator for integration with Ranger Minor fs/azure Thomas Marqardt Thomas Marqardt
HDFS-13183 Standby NameNode process getBlocks request to reduce Active load Major balancer & mover, namenode Xiaoqiao He Xiaoqiao He
HADOOP-17076 ABFS: Delegation SAS Generator Updates Minor fs/azure Thomas Marqardt Thomas Marqardt
HADOOP-15891 Provide Regex Based Mount Point In Inode Tree Major viewfs zhenzhao wang zhenzhao wang
HADOOP-17125 Using snappy-java in SnappyCodec Major common DB Tsai L. C. Hsieh
HADOOP-17292 Using lz4-java in Lz4Codec Major common L. C. Hsieh L. C. Hsieh
HDFS-15711 Add Metrics to HttpFS Server Major httpfs Ahmed Hussein Ahmed Hussein
MAPREDUCE-7315 LocatedFileStatusFetcher to collect/publish IOStatistics Minor client Steve Loughran Steve Loughran
HADOOP-16830 Add Public IOStatistics API Major fs, fs/s3 Steve Loughran Steve Loughran
HDFS-15759 EC: Verify EC reconstruction correctness on DataNode Major datanode, ec, erasure-coding Toshihiko Uchida Toshihiko Uchida
HADOOP-16829 Über-jira: S3A Hadoop 3.3.1 features Major fs/s3 Steve Loughran Steve Loughran

IMPROVEMENTS:

JIRA Summary Priority Component Reporter Contributor
HDFS-15245 Improve JournalNode web UI Major journal-node, ui Jianfei Jiang Jianfei Jiang
HADOOP-16952 Add .diff to gitignore Minor . Ayush Saxena Ayush Saxena
HADOOP-16954 Add -S option in “Count” command to show only Snapshot Counts Major . Hemanth Boyina Hemanth Boyina
HDFS-15247 RBF: Provide Non DFS Used per DataNode in DataNode UI Major . Ayush Saxena Lisheng Sun
MAPREDUCE-7199 HsJobsBlock reuse JobACLsManager for checkAccess Minor . Bibin Chundatt Bilwa S T
HDFS-15295 AvailableSpaceBlockPlacementPolicy should use chooseRandomWithStorageTypeTwoTrial() for better performance. Minor . Jinglun Jinglun
HADOOP-16054 Update Dockerfile to use Bionic Major build, test Akira Ajisaka Akira Ajisaka
YARN-10237 Add isAbsoluteResource config for queue in scheduler response Minor scheduler Prabhu Joseph Prabhu Joseph
HADOOP-16886 Add hadoop.http.idle_timeout.ms to core-default.xml Major . Wei-Chiu Chuang Lisheng Sun
HDFS-14283 DFSInputStream to prefer cached replica Major . Wei-Chiu Chuang Lisheng Sun
HDFS-15338 listOpenFiles() should throw InvalidPathException in case of invalid paths Minor . Jinglun Jinglun
YARN-10160 Add auto queue creation related configs to RMWebService#CapacitySchedulerQueueInfo Major . Prabhu Joseph Prabhu Joseph
HDFS-15255 Consider StorageType when DatanodeManager#sortLocatedBlock() Major . Lisheng Sun Lisheng Sun
YARN-10260 Allow transitioning queue from DRAINING to RUNNING state Major . Jonathan Hung Bilwa S T
HADOOP-17036 TestFTPFileSystem failing as ftp server dir already exists Minor fs, test Steve Loughran Mikhail Pryakhin
HDFS-15356 Unify configuration `dfs.ha.allow.stale.reads` to DFSConfigKeys Major hdfs Xiaoqiao He Xiaoqiao He
HDFS-15358 RBF: Unify router datanode UI with namenode datanode UI Major . Ayush Saxena Ayush Saxena
HADOOP-17042 Hadoop distcp throws “ERROR: Tools helper ///usr/lib/hadoop/libexec/tools/hadoop-distcp.sh was not found” Minor tools/distcp Aki Tanaka Aki Tanaka
HDFS-15202 HDFS-client: boost ShortCircuit Cache Minor dfsclient Danil Lipovoy Danil Lipovoy
HDFS-15207 VolumeScanner skip to scan blocks accessed during recent scan peroid Minor datanode Yang Yun Yang Yun
HDFS-14999 Avoid Potential Infinite Loop in DFSNetworkTopology Major . Ayush Saxena Ayush Saxena
HDFS-13639 SlotReleaser is not fast enough Major hdfs-client Gang Xie Lisheng Sun
HDFS-15369 Refactor method VolumeScanner#runLoop() Minor datanode Yang Yun Yang Yun
HADOOP-14698 Make copyFromLocal’s -t option available for put as well Major . Andras Bokor Andras Bokor
HDFS-10792 RedundantEditLogInputStream should log caught exceptions Minor namenode Wei-Chiu Chuang Wei-Chiu Chuang
YARN-6492 Generate queue metrics for each partition Major capacity scheduler Jonathan Hung Manikandan R
HADOOP-17016 Adding Common Counters in ABFS Major fs/azure Mehakmeet Singh Mehakmeet Singh
HADOOP-16828 Zookeeper Delegation Token Manager fetch sequence number by batch Major . Fengnan Li Fengnan Li
HADOOP-14566 Add seek support for SFTP FileSystem Minor fs Azhagu Selvan SP Mikhail Pryakhin
HADOOP-17047 TODO comments exist in trunk while the related issues are already fixed. Trivial . Rungroj Maipradit Rungroj Maipradit
HADOOP-17020 Improve RawFileSystem Performance Minor fs Rajesh Balamohan Mehakmeet Singh
HDFS-15406 Improve the speed of Datanode Block Scan Major . Hemanth Boyina Hemanth Boyina
HADOOP-17090 Increase precommit job timeout from 5 hours to 20 hours Major build Akira Ajisaka Akira Ajisaka
HADOOP-17084 Update Dockerfile_aarch64 to use Bionic Major build, test RuiChen zhaorenhai
YARN-8047 RMWebApp make external class pluggable Minor . Bibin Chundatt Bilwa S T
YARN-10297 TestContinuousScheduling#testFairSchedulerContinuousSchedulingInitTime fails intermittently Major . Jonathan Hung Jim Brennan
HADOOP-17127 Use RpcMetrics.TIMEUNIT to initialize rpc queueTime and processingTime Minor common Jim Brennan Jim Brennan
HDFS-15404 ShellCommandFencer should expose info about source Major . Chen Liang Chen Liang
HADOOP-17147 Dead link in hadoop-kms/index.md.vm Minor documentation, kms Akira Ajisaka Xieming Li
HADOOP-17113 Adding ReadAhead Counters in ABFS Major fs/azure Mehakmeet Singh Mehakmeet Singh
YARN-10343 Legacy RM UI should include labeled metrics for allocated, total, and reserved resources. Major . Eric Payne Eric Payne
YARN-1529 Add Localization overhead metrics to NM Major nodemanager Gera Shegalov Jim Brennan
YARN-10361 Make custom DAO classes configurable into RMWebApp#JAXBContextResolver Major . Prabhu Joseph Bilwa S T
YARN-10251 Show extended resources on legacy RM UI. Major . Eric Payne Eric Payne
HDFS-15493 Update block map and name cache in parallel while loading fsimage. Major namenode Chengwei Wang Chengwei Wang
HADOOP-17057 ABFS driver enhancement - Allow customizable translation from AAD SPNs and security groups to Linux user and group Major fs/azure Karthik Amarnath Karthik Amarnath
HADOOP-17194 Adding Context class for AbfsClient to pass AbfsConfigurations to limit number of parameters Major fs/azure Mehakmeet Singh Mehakmeet Singh
HADOOP-17065 Adding Network Counters in ABFS Major fs/azure Mehakmeet Singh Mehakmeet Singh
HADOOP-17159 Make UGI support forceful relogin from keytab ignoring the last login time Major security Sandeep Guggilam Sandeep Guggilam
YARN-10407 Add phantomjsdriver.log to gitignore Minor . Takanobu Asanuma Takanobu Asanuma
YARN-10353 Log vcores used and cumulative cpu in containers monitor Minor yarn Jim Brennan Jim Brennan
YARN-10369 Make NMTokenSecretManagerInRM sending NMToken for nodeId DEBUG Minor yarn Jim Brennan Jim Brennan
YARN-10390 LeafQueue: retain user limits cache across assignContainers() calls Major capacity scheduler, capacityscheduler Muhammad Samir Khan Muhammad Samir Khan
HDFS-15574 Remove unnecessary sort of block list in DirectoryScanner Major . Stephen O’Donnell Stephen O’Donnell
HADOOP-17270 Fix testCompressorDecompressorWithExeedBufferLimit to cover the intended scenario Minor test Masatake Iwasaki Masatake Iwasaki
HDFS-15581 Access Controlled HTTPFS Proxy Minor httpfs Richard Richard
HADOOP-17283 Hadoop - Upgrade to JQuery 3.5.1 Major . Aryan Gupta Aryan Gupta
HADOOP-17267 Add debug-level logs in Filesystem#close Minor fs Karen Coppage Karen Coppage
HADOOP-17284 Support BCFKS keystores for Hadoop Credential Provider Major . Xiaoyu Yao Xiaoyu Yao
HDFS-15415 Reduce locking in Datanode DirectoryScanner Major datanode Stephen O’Donnell Stephen O’Donnell
YARN-10451 RM (v1) UI NodesPage can NPE when yarn.io/gpu resource type is defined. Major . Eric Payne Eric Payne
HADOOP-17021 Add concat fs command Minor fs Jinglun Jinglun
MAPREDUCE-7301 Expose Mini MR Cluster attribute for testing Minor test Swaroopa Kadam Swaroopa Kadam
HDFS-15567 [SBN Read] HDFS should expose msync() API to allow downstream applications call it explicitly. Major ha, hdfs-client Konstantin Shvachko Konstantin Shvachko
HDFS-15633 Avoid redundant RPC calls for getDiskStatus Major dfsclient Ayush Saxena Ayush Saxena
YARN-10450 Add cpu and memory utilization per node and cluster-wide metrics Minor yarn Jim Brennan Jim Brennan
HADOOP-17302 Upgrade to jQuery 3.5.1 in hadoop-sls Major . Aryan Gupta Aryan Gupta
HDFS-15652 Make block size from NNThroughputBenchmark configurable Minor benchmarks Hui Fei Hui Fei
YARN-10475 Scale RM-NM heartbeat interval based on node utilization Minor yarn Jim Brennan Jim Brennan
HDFS-15665 Balancer logging improvement Major balancer & mover Konstantin Shvachko Konstantin Shvachko
HADOOP-17342 Creating a token identifier should not do kerberos name resolution Major common Jim Brennan Jim Brennan
YARN-10479 RMProxy should retry on SocketTimeout Exceptions Major yarn Jim Brennan Jim Brennan
HDFS-15623 Respect configured values of rpc.engine Major hdfs Hector Sandoval Chaverri Hector Sandoval Chaverri
HADOOP-17369 Bump up snappy-java to 1.1.8.1 Minor . Masatake Iwasaki Masatake Iwasaki
YARN-10480 replace href tags with ng-href Trivial . Gabriel Medeiros Coelho Gabriel Medeiros Coelho
HADOOP-17367 Add InetAddress api to ProxyUsers.authorize Major performance, security Ahmed Hussein Ahmed Hussein
MAPREDUCE-7304 Enhance the map-reduce Job end notifier to be able to notify the given URL via a custom class Major mrv2 Daniel Fritsi Zoltán Erdmann
MAPREDUCE-7309 Improve performance of reading resource request for mapper/reducers from config Major applicationmaster Wangda Tan Peter Bacsko
HDFS-15694 Avoid calling UpdateHeartBeatState inside DataNodeDescriptor Major . Ahmed Hussein Ahmed Hussein
HDFS-15703 Don’t generate edits for set operations that are no-op Major namenode Ahmed Hussein Ahmed Hussein
HADOOP-17392 Remote exception messages should not include the exception class Major ipc Ahmed Hussein Ahmed Hussein
HDFS-15706 HttpFS: Log more information on request failures Major httpfs Ahmed Hussein Ahmed Hussein
HADOOP-17389 KMS should log full UGI principal Major . Ahmed Hussein Ahmed Hussein
HADOOP-17425 Bump up snappy-java to 1.1.8.2 Minor . L. C. Hsieh L. C. Hsieh
HDFS-15720 namenode audit async logger should add some log4j config Minor hdfs Max Xie
HDFS-15717 Improve fsck logging Major logging, namenode Ahmed Hussein Ahmed Hussein
HDFS-15704 Mitigate lease monitor’s rapid infinite loop Major namenode Ahmed Hussein Ahmed Hussein
HDFS-15569 Speed up the Storage#doRecover during datanode rolling upgrade Major . Hemanth Boyina Hemanth Boyina
HDFS-15751 Add documentation for msync() API to filesystem.md Major documentation Konstantin Shvachko Konstantin Shvachko
HADOOP-17454 [s3a] Disable bucket existence check - set fs.s3a.bucket.probe to 0 Major . Gabor Bota Gabor Bota
YARN-10538 Add recommissioning nodes to the list of updated nodes returned to the AM Major . Srinivas S T Srinivas S T
YARN-10541 capture the performance metrics of ZKRMStateStore Minor resourcemanager Max Xie Max Xie
HADOOP-17408 Optimize NetworkTopology while sorting of block locations Major common, net Ahmed Hussein Ahmed Hussein
YARN-4589 Diagnostics for localization timeouts is lacking Major . Chang Li Chang Li
YARN-10562 Follow up changes for YARN-9833 Major yarn Jim Brennan Jim Brennan
HDFS-15783 Speed up BlockPlacementPolicyRackFaultTolerant#verifyBlockPlacement Major block placement Akira Ajisaka Akira Ajisaka
YARN-10519 Refactor QueueMetricsForCustomResources class to move to yarn-common package Major . Minni Mittal Minni Mittal
HADOOP-17484 Typo in hadop-aws index.md Trivial documentation, fs/s3 Maksim Maksim
HADOOP-17478 Improve the description of hadoop.http.authentication.signature.secret.file Minor documentation Akira Ajisaka Akira Ajisaka
MAPREDUCE-7317 Add latency information in FileOutputCommitter.mergePaths Minor client Jungtaek Lim Jungtaek Lim
HDFS-15789 Lease renewal does not require namesystem lock Major hdfs Jim Brennan Jim Brennan
HADOOP-17501 Fix logging typo in ShutdownHookManager Major common Konstantin Shvachko Fengnan Li
HADOOP-17354 Move Jenkinsfile outside of the root directory Major build Akira Ajisaka Akira Ajisaka
HADOOP-17508 Simplify dependency installation instructions Trivial documentation Gautham Banasandra Gautham Banasandra
HADOOP-17509 Parallelize building of dependencies Minor build Gautham Banasandra Gautham Banasandra
HDFS-15799 Make DisallowedDatanodeException terse Minor hdfs Richard Richard
HDFS-15813 DataStreamer: keep sending heartbeat packets while streaming Major hdfs Jim Brennan Jim Brennan
MAPREDUCE-7319 Log list of mappers at trace level in ShuffleHandler audit log Minor yarn Jim Brennan Jim Brennan
HADOOP-14402 roll out StreamCapabilities across output streams of all filesystems Major fs, fs/adl, fs/azure, fs/oss, fs/s3, fs/swift Steve Loughran Steve Loughran
HDFS-15821 Add metrics for in-service datanodes Minor . Zehao Chen Zehao Chen
YARN-10626 Log resource allocation in NM log at container start time Major . Eric Badger Eric Badger
HDFS-15815 if required storageType are unavailable, log the failed reason during choosing Datanode Minor block placement Yang Yun Yang Yun
HDFS-15830 Support to make dfs.image.parallel.load reconfigurable Major namenode Hui Fei Hui Fei
HDFS-15835 Erasure coding: Add/remove logs for the better readability/debugging Minor erasure-coding, hdfs Bhavik Patel Bhavik Patel
HDFS-15826 Solve the problem of incorrect progress of delegation tokens when loading FsImage Major . JiangHua Zhu JiangHua Zhu
HDFS-15734 [READ] DirectoryScanner#scan need not check StorageType.PROVIDED Minor datanode Yuxuan Wang Yuxuan Wang
HADOOP-17538 Add kms-default.xml and httpfs-default.xml to site index Minor documentation Masatake Iwasaki Masatake Iwasaki
YARN-10613 Config to allow Intra- and Inter-queue preemption to enable/disable conservativeDRF Minor capacity scheduler, scheduler preemption Eric Payne Eric Payne
YARN-10653 Fixed the findbugs issues introduced by YARN-10647. Major . Qi Zhu Qi Zhu
MAPREDUCE-7324 ClientHSSecurityInfo class is in wrong META-INF file Major . Eric Badger Eric Badger
HADOOP-17546 Update Description of hadoop-http-auth-signature-secret in HttpAuthentication.md Minor . Ravuri Sushma sree Ravuri Sushma sree
YARN-10664 Allow parameter expansion in NM_ADMIN_USER_ENV Major yarn Jim Brennan Jim Brennan
HADOOP-17570 Apply YETUS-1102 to re-enable GitHub comments Major build Akira Ajisaka Akira Ajisaka
HADOOP-17594 DistCp: Expose the JobId for applications executing through run method Major . Ayush Saxena Ayush Saxena
HDFS-15907 Reduce Memory Overhead of AclFeature by avoiding AtomicInteger Major namenode Stephen O’Donnell Stephen O’Donnell
HDFS-15911 Provide blocks moved count in Balancer iteration result Major balancer & mover Viraj Jasani Viraj Jasani
HDFS-15919 BlockPoolManager should log stack trace if unable to get Namenode addresses Major datanode Stephen O’Donnell Stephen O’Donnell
HADOOP-17531 DistCp: Reduce memory usage on copying huge directories Critical . Ayush Saxena Ayush Saxena
HDFS-15879 Exclude slow nodes when choose targets for blocks Major . tomscut tomscut
HADOOP-16870 Use spotbugs-maven-plugin instead of findbugs-maven-plugin Major build Akira Ajisaka Akira Ajisaka
HADOOP-17222 Create socket address leveraging URI cache Major common, hdfs-client fanrui fanrui
HDFS-15932 Improve the balancer error message when process exits abnormally. Major . Renukaprasad C Renukaprasad C
HADOOP-16524 Automatic keystore reloading for HttpServer2 Major . Kihwal Lee Borislav Iordanov
HDFS-15931 Fix non-static inner classes for better memory management Major . Viraj Jasani Viraj Jasani
HADOOP-17371 Bump Jetty to the latest version 9.4.35 Major . Wei-Chiu Chuang Wei-Chiu Chuang
HDFS-15942 Increase Quota initialization threads Major namenode Stephen O’Donnell Stephen O’Donnell
HADOOP-17613 Log not flushed fully when daemon shutdown Major common Renukaprasad C Renukaprasad C
HDFS-15937 Reduce memory used during datanode layout upgrade Major datanode Stephen O’Donnell Stephen O’Donnell
HDFS-15160 ReplicaMap, Disk Balancer, Directory Scanner and various FsDatasetImpl methods should use datanode readlock Major datanode Stephen O’Donnell Stephen O’Donnell
HADOOP-17569 Building native code fails on Fedora 33 Major build, common Kengo Seki Masatake Iwasaki
HADOOP-17633 Please upgrade json-smart dependency to the latest version Major auth, build helen huang Viraj Jasani
HADOOP-17620 DistCp: Use Iterator for listing target directory as well Major . Ayush Saxena Ayush Saxena
YARN-10743 Add a policy for not aggregating for containers which are killed because exceeding container log size limit. Major . Qi Zhu Qi Zhu
HDFS-15967 Improve the log for Short Circuit Local Reads Minor . Bhavik Patel Bhavik Patel
HADOOP-17675 LdapGroupsMapping$LdapSslSocketFactory ClassNotFoundException Major common Tamas Mate István Fajth
HADOOP-11616 Remove workaround for Curator’s ChildReaper requiring Guava 15+ Major . Robert Kanter Viraj Jasani
HDFS-16003 ProcessReport print invalidatedBlocks should judge debug level at first Minor namanode lei w lei w
HDFS-16007 Deserialization of ReplicaState should avoid throwing ArrayIndexOutOfBoundsException Major . junwen yang Viraj Jasani
HADOOP-17615 ADLFS: Update SDK version from 2.3.6 to 2.3.9 Minor fs/adl Bilahari T H Bilahari T H
HADOOP-16822 Provide source artifacts for hadoop-client-api Major . Karel Kolman Karel Kolman
HADOOP-17680 Allow ProtobufRpcEngine to be extensible Major common Hector Sandoval Chaverri Hector Sandoval Chaverri
YARN-10258 Add metrics for ‘ApplicationsRunning’ in NodeManager Minor nodemanager ANANDA G B ANANDA G B
HDFS-15790 Make ProtobufRpcEngineProtos and ProtobufRpcEngineProtos2 Co-Exist Critical . David Mollitor Vinayakumar B

BUG FIXES:

JIRA Summary Priority Component Reporter Contributor
HDFS-15196 RBF: RouterRpcServer getListing cannot list large dirs correctly Critical . Fengnan Li Fengnan Li
HDFS-15252 HttpFS: setWorkingDirectory should not accept invalid paths Major . Hemanth Boyina Hemanth Boyina
HDFS-15249 ThrottledAsyncChecker is not thread-safe. Major federation Toshihiro Suzuki Toshihiro Suzuki
HDFS-15266 Add missing DFSOps Statistics in WebHDFS Major . Ayush Saxena Ayush Saxena
HDFS-15275 HttpFS: Response of Create was not correct with noredirect and data are true Major . Hemanth Boyina Hemanth Boyina
HDFS-15281 ZKFC ignores dfs.namenode.rpc-bind-host and uses dfs.namenode.rpc-address to bind to host address Major ha, namenode Dhiraj Hegde Dhiraj Hegde
HDFS-15297 TestNNHandlesBlockReportPerStorage::blockReport_02 fails intermittently in trunk Major datanode, test Mingliang Liu Ayush Saxena
HDFS-15210 EC : File write hanged when DN is shutdown by admin command. Major ec Surendra Singh Lilhore Surendra Singh Lilhore
HDFS-15285 The same distance and load nodes don’t shuffle when consider DataNode load Major . Lisheng Sun Lisheng Sun
HDFS-15265 HttpFS: validate content-type in HttpFSUtils Major . Hemanth Boyina Hemanth Boyina
HDFS-15320 StringIndexOutOfBoundsException in HostRestrictingAuthorizationFilter Major webhdfs Akira Ajisaka Akira Ajisaka
YARN-10256 Refactor TestContainerSchedulerQueuing.testContainerUpdateExecTypeGuaranteedToOpportunistic Major . Ahmed Hussein Ahmed Hussein
HDFS-15270 Account for *env == NULL in hdfsThreadDestructor Major . Babneet Singh Babneet Singh
HDFS-15331 Remove invalid exclusions that minicluster dependency on HDFS Major . Wanqiang Ji Wanqiang Ji
YARN-8959 TestContainerResizing fails randomly Minor . Bibin Chundatt Ahmed Hussein
HDFS-15332 Quota Space consumed was wrong in truncate with Snapshots Major . Hemanth Boyina Hemanth Boyina
YARN-9017 PlacementRule order is not maintained in CS Major . Bibin Chundatt Bilwa S T
HADOOP-17025 Fix invalid metastore configuration in S3GuardTool tests Minor fs/s3, test Masatake Iwasaki Masatake Iwasaki
HDFS-15339 TestHDFSCLI fails for user names with the dot/dash character Major test Yan Xiaole Yan Xiaole
HDFS-15250 Setting `dfs.client.use.datanode.hostname` to true can crash the system because of unhandled UnresolvedAddressException Major . Ctest Ctest
HADOOP-16768 SnappyCompressor test cases wrongly assume that the compressed data is always smaller than the input data Major io, test zhao bo Akira Ajisaka
HDFS-1820 FTPFileSystem attempts to close the outputstream even when it is not initialised Major hdfs-client Sudharsan Sampath Mikhail Pryakhin
HDFS-15243 Add an option to prevent sub-directories of protected directories from deletion Major 3.1.1 liuyanyu liuyanyu
HDFS-14367 EC: Parameter maxPoolSize in striped reconstruct thread pool isn’t affecting number of threads Major ec Guo Lei Guo Lei
YARN-9301 Too many InvalidStateTransitionException with SLS Major . Bibin Chundatt Bilwa S T
HDFS-15300 RBF: updateActiveNamenode() is invalid when RPC address is IP Major . xuzq xuzq
HDFS-15316 Deletion failure should not remove directory from snapshottables Major . Hemanth Boyina Hemanth Boyina
YARN-8942 PriorityBasedRouterPolicy throws exception if all sub-cluster weights have negative value Minor . Akshay Agarwal Bilwa S T
HADOOP-17044 Revert “HADOOP-8143. Change distcp to have -pb on by default” Major tools/distcp Steve Loughran Steve Loughran
HADOOP-17024 ListStatus on ViewFS root (ls “/”) should list the linkFallBack root (configured target root). Major fs, viewfs Uma Maheswara Rao G Abhishek Das
MAPREDUCE-6826 Job fails with InvalidStateTransitonException: Invalid event: JOB_TASK_COMPLETED at SUCCEEDED/COMMITTING Major . Varun Saxena Bilwa S T
HADOOP-16900 Very large files can be truncated when written through S3AFileSystem Major fs/s3 Andrew Olson Mukund Thakur
HADOOP-17049 javax.activation-api and jakarta.activation-api define overlapping classes Major build Akira Ajisaka Akira Ajisaka
HADOOP-17040 Fix intermittent failure of ITestBlockingThreadPoolExecutorService Minor fs/s3, test Masatake Iwasaki Masatake Iwasaki
HDFS-15363 BlockPlacementPolicyWithNodeGroup should validate if it is initialized by NetworkTopologyWithNodeGroup Major . Hemanth Boyina Hemanth Boyina
HDFS-15093 RENAME.TO_TRASH is ignored When RENAME.OVERWRITE is specified Major . Harshakiran Reddy Ayush Saxena
HDFS-12288 Fix DataNode’s xceiver count calculation Major datanode, hdfs Lukas Majercak Lisheng Sun
HDFS-15362 FileWithSnapshotFeature#updateQuotaAndCollectBlocks should collect all distinct blocks Major . Hemanth Boyina Hemanth Boyina
HADOOP-7002 Wrong description of copyFromLocal and copyToLocal in documentation Minor . Jingguo Yao Andras Bokor
HADOOP-17052 NetUtils.connect() throws unchecked exception (UnresolvedAddressException) causing clients to abort Major net Dhiraj Hegde Dhiraj Hegde
HADOOP-17018 Intermittent failing of ITestAbfsStreamStatistics in ABFS Minor fs/azure, test Mehakmeet Singh Mehakmeet Singh
YARN-10254 CapacityScheduler incorrect User Group Mapping after leaf queue change Major . Gergely Pollák Gergely Pollák
HADOOP-17062 Fix shelldocs path in Jenkinsfile Major build Akira Ajisaka Akira Ajisaka
HADOOP-17056 shelldoc fails in hadoop-common Major build Akira Ajisaka Akira Ajisaka
YARN-10286 PendingContainers bugs in the scheduler outputs Critical . Adam Antal Andras Gyori
HDFS-15396 Fix TestViewFileSystemOverloadSchemeHdfsFileSystemContract#testListStatusRootDir Major . Ayush Saxena Ayush Saxena
HDFS-15386 ReplicaNotFoundException keeps happening in DN after removing multiple DN’s data directories Major . Toshihiro Suzuki Toshihiro Suzuki
HDFS-15398 EC: hdfs client hangs due to exception during addBlock Critical ec, hdfs-client Hongbing Wang Hongbing Wang
YARN-10300 appMasterHost not set in RM ApplicationSummary when AM fails before first heartbeat Major . Eric Badger Eric Badger
HADOOP-17059 ArrayIndexOfboundsException in ViewFileSystem#listStatus Major viewfs Hemanth Boyina Hemanth Boyina
YARN-10296 Make ContainerPBImpl#getId/setId synchronized Minor . Benjamin Teke Benjamin Teke
HADOOP-17060 listStatus and getFileStatus behave inconsistent in the case of ViewFs implementation for isDirectory Major viewfs Srinivasu Majeti Uma Maheswara Rao G
YARN-10312 Add support for yarn logs -logFile to retain backward compatibility Major client Jim Brennan Jim Brennan
HDFS-15351 Blocks scheduled count was wrong on truncate Major . Hemanth Boyina Hemanth Boyina
HDFS-15403 NPE in FileIoProvider#transferToSocketFully Major . Hemanth Boyina Hemanth Boyina
HDFS-15372 Files in snapshots no longer see attribute provider permissions Major . Stephen O’Donnell Stephen O’Donnell
MAPREDUCE-7281 Fix NoClassDefFoundError on ‘mapred minicluster’ Major . Masatake Iwasaki Masatake Iwasaki
HADOOP-17029 ViewFS does not return correct user/group and ACL Major fs, viewfs Abhishek Das Abhishek Das
HDFS-14546 Document block placement policies Major . Íñigo Goiri Amithsha
HADOOP-17068 client fails forever when namenode ipaddr changed Major hdfs-client Sean Chow Sean Chow
HADOOP-17089 WASB: Update azure-storage-java SDK Critical fs/azure Thomas Marqardt Thomas Marqardt
HDFS-15378 TestReconstructStripedFile#testErasureCodingWorkerXmitsWeight is failing on trunk Major . Hemanth Boyina Hemanth Boyina
YARN-9903 Support reservations continue looking for Node Labels Major . Tarun Parimi Jim Brennan
YARN-10331 Upgrade node.js to 10.21.0 Critical build, yarn-ui-v2 Akira Ajisaka Akira Ajisaka
HADOOP-17032 Handle an internal dir in viewfs having multiple children mount points pointing to different filesystems Major fs, viewfs Abhishek Das Abhishek Das
YARN-10318 ApplicationHistory Web UI incorrect column indexing Minor yarn Andras Gyori Andras Gyori
YARN-10330 Add missing test scenarios to TestUserGroupMappingPlacementRule and TestAppNameMappingPlacementRule Major capacity scheduler, capacityscheduler, test Peter Bacsko Peter Bacsko
HDFS-15446 CreateSnapshotOp fails during edit log loading for /.reserved/raw/path with error java.io.FileNotFoundException: Directory does not exist: /.reserved/raw/path Major hdfs Srinivasu Majeti Stephen O’Donnell
HADOOP-17081 MetricsSystem doesn’t start the sink adapters on restart Minor metrics Madhusoodan Madhusoodan
HDFS-15451 Restarting name node stuck in safe mode when using provided storage Major namenode shanyu zhao shanyu zhao
HADOOP-17117 Fix typos in hadoop-aws documentation Trivial documentation, fs/s3 Sebastian Nagel Sebastian Nagel
HADOOP-17120 Fix failure of docker image creation due to pip2 install error Major . Masatake Iwasaki Masatake Iwasaki
YARN-10344 Sync netty versions in hadoop-yarn-csi Major build Akira Ajisaka Akira Ajisaka
YARN-10341 Yarn Service Container Completed event doesn’t get processed Critical . Bilwa S T Bilwa S T
HADOOP-16998 WASB : NativeAzureFsOutputStream#close() throwing IllegalArgumentException Major fs/azure Anoop Sam John Anoop Sam John
YARN-10348 Allow RM to always cancel tokens after app completes Major yarn Jim Brennan Jim Brennan
MAPREDUCE-7284 TestCombineFileInputFormat#testMissingBlocks fails Major test Akira Ajisaka Akira Ajisaka
HDFS-14498 LeaseManager can loop forever on the file for which create has failed Major namenode Sergey Shelukhin Stephen O’Donnell
HADOOP-17130 Configuration.getValByRegex() shouldn’t update the results while fetching. Major common Mukund Thakur Mukund Thakur
HDFS-15198 RBF: Add test for MountTableRefresherService failed to refresh other router MountTableEntries in secure mode Major rbf zhengchenyu zhengchenyu
HADOOP-17119 Jetty upgrade to 9.4.x causes MR app fail with IOException Major . Bilwa S T Bilwa S T
HDFS-15246 ArrayIndexOfboundsException in BlockManager CreateLocatedBlock Major . Hemanth Boyina Hemanth Boyina
HADOOP-17138 Fix spotbugs warnings surfaced after upgrade to 4.0.6 Minor . Masatake Iwasaki Masatake Iwasaki
YARN-4771 Some containers can be skipped during log aggregation after NM restart Major nodemanager Jason Darrell Lowe Jim Brennan
YARN-10367 Failed to get nodejs 10.21.0 when building docker image Blocker build, webapp Akira Ajisaka Akira Ajisaka
MAPREDUCE-7051 Fix typo in MultipleOutputFormat Trivial . ywheel ywheel
HDFS-15313 Ensure inodes in active filesystem are not deleted during snapshot delete Major snapshots Shashikant Banerjee Shashikant Banerjee
HDFS-14950 missing libhdfspp libs in dist-package Major build, libhdfs++ Yuan Zhou Yuan Zhou
YARN-10359 Log container report only if list is not empty Minor . Bilwa S T Bilwa S T
YARN-10229 [Federation] Client should be able to submit application to RM directly using normal client conf Major amrmproxy, federation JohnsonGuo Bilwa S T
HDFS-15503 File and directory permissions are not able to be modified from WebUI Major . Hemanth Boyina Hemanth Boyina
HADOOP-17184 Add –mvn-custom-repos parameter to yetus calls Major build Mingliang Liu Mingliang Liu
HDFS-15499 Clean up httpfs/pom.xml to remove aws-java-sdk-s3 exclusion Major httpfs Mingliang Liu Mingliang Liu
HADOOP-17186 Fixing javadoc in ListingOperationCallbacks Major build, documentation Akira Ajisaka Mukund Thakur
HADOOP-17164 UGI loginUserFromKeytab doesn’t set the last login time Major security Sandeep Guggilam Sandeep Guggilam
YARN-4575 ApplicationResourceUsageReport should return ALL reserved resource Major . Bibin Chundatt Bibin Chundatt
YARN-10388 RMNode updatedCapability flag not set while RecommissionNodeTransition Major resourcemanager Pranjal Protim Borah Pranjal Protim Borah
HDFS-15443 Setting dfs.datanode.max.transfer.threads to a very small value can cause strange failure. Major datanode AMC-team AMC-team
HDFS-15508 [JDK 11] Fix javadoc errors in hadoop-hdfs-rbf module Major documentation Akira Ajisaka Akira Ajisaka
HDFS-15506 [JDK 11] Fix javadoc errors in hadoop-hdfs module Major documentation Akira Ajisaka Xieming Li
HDFS-15507 [JDK 11] Fix javadoc errors in hadoop-hdfs-client module Major documentation Akira Ajisaka Xieming Li
HADOOP-17196 Fix C/C++ standard warnings Major build Gautham Banasandra Gautham Banasandra
HADOOP-17204 Fix typo in Hadoop KMS document Trivial documentation, kms Akira Ajisaka Xieming Li
HADOOP-17192 ITestS3AHugeFilesSSECDiskBlock failing because of bucket overrides Major fs/s3 Mukund Thakur Mukund Thakur
YARN-10336 RM page should throw exception when command injected in RM REST API to get applications Major . Rajshree Mishra Bilwa S T
HDFS-15439 Setting dfs.mover.retry.max.attempts to negative value will retry forever. Major balancer & mover AMC-team AMC-team
YARN-10391 –module-gpu functionality is broken in container-executor Major nodemanager Eric Badger Eric Badger
HADOOP-17122 Bug in preserving Directory Attributes in DistCp with Atomic Copy Major tools/distcp Swaminathan Balachandran
HDFS-14504 Rename with Snapshots does not honor quota limit Major . Shashikant Banerjee Hemanth Boyina
HADOOP-17209 Erasure Coding: Native library memory leak Major native Sean Chow Sean Chow
HADOOP-16925 MetricsConfig incorrectly loads the configuration whose value is String list in the properties file Major metrics Jiayi Liu Jiayi Liu
HADOOP-17220 Upgrade slf4j to 1.7.30 ( To Address: CVE-2018-8088) Major . Brahma Reddy Battula Brahma Reddy Battula
HDFS-14852 Removing from LowRedundancyBlocks does not remove the block from all queues Major namenode Hui Fei Hui Fei
HDFS-15536 RBF: Clear Quota in Router was not consistent Critical . Hemanth Boyina Hemanth Boyina
HDFS-15510 RBF: Quota and Content Summary was not correct in Multiple Destinations Critical . Hemanth Boyina Hemanth Boyina
HDFS-15540 Directories protected from delete can still be moved to the trash Major namenode Stephen O’Donnell Stephen O’Donnell
HADOOP-17129 Validating storage keys in ABFS correctly Major fs/azure Mehakmeet Singh Mehakmeet Singh
HDFS-15471 TestHDFSContractMultipartUploader fails on trunk Major test Ahmed Hussein Steve Loughran
HDFS-15290 NPE in HttpServer during NameNode startup Major namenode Konstantin Shvachko Simbarashe Dzinamarira
HADOOP-17158 Test timeout for ITestAbfsInputStreamStatistics#testReadAheadCounters Major fs/azure Mehakmeet Singh Mehakmeet Singh
HADOOP-17229 Test failure as failed request body counted in byte received metric - ITestAbfsNetworkStatistics#testAbfsHttpResponseStatistics Major fs/azure, test Sneha Vijayarajan Mehakmeet Singh
YARN-10397 SchedulerRequest should be forwarded to scheduler if custom scheduler supports placement constraints Minor . Bilwa S T Bilwa S T
HDFS-15573 Only log warning if considerLoad and considerStorageType are both true Major . Stephen O’Donnell Stephen O’Donnell
YARN-10430 Log improvements in NodeStatusUpdaterImpl Minor nodemanager Bilwa S T Bilwa S T
HADOOP-17246 Fix build the hadoop-build Docker image failed Major build Wanqiang Ji Wanqiang Ji
HDFS-15438 Setting dfs.disk.balancer.max.disk.errors = 0 will fail the block copy Major balancer & mover AMC-team AMC-team
HADOOP-15136 Typo in rename spec pseudocode Major documentation Rae Marks
HADOOP-17088 Failed to load XInclude files with relative path. Minor conf Yushi Hayasaka Yushi Hayasaka
MAPREDUCE-7294 Only application master should upload resource to Yarn Shared Cache Major mrv2 zhenzhao wang zhenzhao wang
HADOOP-17277 Correct spelling errors for separator Trivial common Hui Fei Hui Fei
HADOOP-17286 Upgrade to jQuery 3.5.1 in hadoop-yarn-common Major . Wei-Chiu Chuang Aryan Gupta
HDFS-15591 RBF: Fix webHdfs file display error Major . wangzhaohui wangzhaohui
MAPREDUCE-7289 Fix wrong comment in LongLong.java Trivial documentation, examples Akira Ajisaka Wanqiang Ji
YARN-9809 NMs should supply a health status when registering with RM Major . Eric Badger Eric Badger
HADOOP-17300 FileSystem.DirListingIterator.next() call should return NoSuchElementException Major common, fs Mukund Thakur Mukund Thakur
YARN-10393 MR job live lock caused by completed state container leak in heartbeat between node manager and RM Major nodemanager, yarn zhenzhao wang Jim Brennan
HDFS-15253 Set default throttle value on dfs.image.transfer.bandwidthPerSec Major namenode Karthik Palanisamy Karthik Palanisamy
HDFS-15610 Reduce datanode upgrade/hardlink thread Major datanode Karthik Palanisamy Karthik Palanisamy
YARN-10455 TestNMProxy.testNMProxyRPCRetry is not consistent Major . Ahmed Hussein Ahmed Hussein
HDFS-15456 TestExternalStoragePolicySatisfier fails intermittently Major . Ahmed Hussein Leon Gao
HADOOP-17223 update org.apache.httpcomponents:httpclient to 4.5.13 and httpcore to 4.4.13 Blocker . Pranav Bheda Pranav Bheda
HDFS-15628 HttpFS server throws NPE if a file is a symlink Major fs, httpfs Ahmed Hussein Ahmed Hussein
HDFS-15627 Audit log deletes before collecting blocks Major logging, namenode Ahmed Hussein Ahmed Hussein
HADOOP-17309 Javadoc warnings and errors are ignored in the precommit jobs Major build, documentation Akira Ajisaka Akira Ajisaka
HDFS-14383 Compute datanode load based on StoragePolicy Major hdfs, namenode Karthik Palanisamy Ayush Saxena
HADOOP-17310 Touch command with -c option is broken Major . Ayush Saxena Ayush Saxena
HDFS-15626 TestWebHDFS.testLargeDirectory failing Major test, webhdfs Mukund Thakur Mukund Thakur
HADOOP-17298 Backslash in username causes build failure in the environment started by start-build-env.sh. Minor build Takeru Kuramoto Takeru Kuramoto
HDFS-15639 [JDK 11] Fix Javadoc errors in hadoop-hdfs-client Major . Takanobu Asanuma Takanobu Asanuma
HDFS-15622 Deleted blocks linger in the replications queue Major hdfs Ahmed Hussein Ahmed Hussein
HADOOP-17308 WASB : PageBlobOutputStream succeeding hflush even when underlying flush to storage failed Critical . Anoop Sam John Anoop Sam John
HDFS-15641 DataNode could meet deadlock if invoke refreshNameNode Critical . Hongbing Wang Hongbing Wang
HADOOP-17328 LazyPersist Overwrite fails in direct write mode Major . Ayush Saxena Ayush Saxena
MAPREDUCE-7302 Upgrading to JUnit 4.13 causes testcase TestFetcher.testCorruptedIFile() to fail Major test Peter Bacsko Peter Bacsko
HDFS-15644 Failed volumes can cause DNs to stop block reporting Major block placement, datanode Ahmed Hussein Ahmed Hussein
HADOOP-17236 Bump up snakeyaml to 1.26 to mitigate CVE-2017-18640 Major . Brahma Reddy Battula Brahma Reddy Battula
YARN-10467 ContainerIdPBImpl objects can be leaked in RMNodeImpl.completedContainers Major resourcemanager Haibo Chen Haibo Chen
HADOOP-17329 mvn site commands fails due to MetricsSystemImpl changes Major . Xiaoqiao He Xiaoqiao He
HDFS-15651 Client could not obtain block when DN CommandProcessingThread exit Major . Yiqun Lin Aiphago
HADOOP-17340 TestLdapGroupsMapping failing -string mismatch in exception validation Major test Steve Loughran Steve Loughran
HDFS-15667 Audit log record the unexpected allowed result when delete called Major hdfs Baolong Mao Baolong Mao
HADOOP-17352 Update PATCH_NAMING_RULE in the personality file Minor build Akira Ajisaka Akira Ajisaka
YARN-10458 Hive On Tez queries fails upon submission to dynamically created pools Major resourcemanager Anand Srinivasan Peter Bacsko
HDFS-15485 Fix outdated properties of JournalNode when performing rollback Minor . Deegue Deegue
HADOOP-17096 ZStandardCompressor throws java.lang.InternalError: Error (generic) Major io Stephen Jung (Stripe) Stephen Jung (Stripe)
HADOOP-17327 NPE when starting MiniYARNCluster from hadoop-client-minicluster Critical . Chao Sun
HADOOP-17324 Don’t relocate org.bouncycastle in shaded client jars Critical . Chao Sun Chao Sun
HADOOP-17373 hadoop-client-integration-tests doesn’t work when building with skipShade Major . Chao Sun Chao Sun
HADOOP-17365 Contract test for renaming over existing file is too lenient Minor test Attila Doroszlai Attila Doroszlai
HADOOP-17358 Improve excessive reloading of Configurations Major conf Ahmed Hussein Ahmed Hussein
HDFS-15538 Fix the documentation for dfs.namenode.replication.max-streams in hdfs-default.xml Major . Xieming Li Xieming Li
HADOOP-17362 Doing hadoop ls on Har file triggers too many RPC calls Major fs Ahmed Hussein Ahmed Hussein
YARN-10485 TimelineConnector swallows InterruptedException Major . Ahmed Hussein Ahmed Hussein
HADOOP-17360 Log the remote address for authentication success Minor ipc Ahmed Hussein Ahmed Hussein
HDFS-15685 [JDK 14] TestConfiguredFailoverProxyProvider#testResolveDomainNameUsingDNS fails Major . Akira Ajisaka Akira Ajisaka
MAPREDUCE-7305 [JDK 11] TestMRJobsWithProfiler fails Major test Akira Ajisaka Akira Ajisaka
YARN-10396 Max applications calculation per queue disregards queue level settings in absolute mode Major capacity scheduler Benjamin Teke Benjamin Teke
HADOOP-17390 Skip license check on lz4 code files Major build Zhihua Deng Zhihua Deng
MAPREDUCE-7307 Potential thread leak in LocatedFileStatusFetcher Major job submission Zhihua Deng Zhihua Deng
HADOOP-17346 Fair call queue is defeated by abusive service principals Major common, ipc Ahmed Hussein Ahmed Hussein
YARN-10470 When building new web ui with root user, the bower install should support it. Major build, yarn-ui-v2 Qi Zhu Qi Zhu
HADOOP-17398 Skipping network I/O in S3A getFileStatus(/) breaks some tests Major fs/s3, test Mukund Thakur Mukund Thakur
HDFS-15698 Fix the typo of dfshealth.html after HDFS-15358 Trivial namenode Hui Fei Hui Fei
YARN-10498 Fix Yarn CapacityScheduler Markdown document Trivial documentation zhaoshengjie zhaoshengjie
HADOOP-17399 lz4 sources missing for native Visual Studio project Major native Gautham Banasandra Gautham Banasandra
HDFS-15695 NN should not let the balancer run in safemode Major namenode Ahmed Hussein Ahmed Hussein
YARN-10511 Update yarn.nodemanager.env-whitelist value in docs Minor documentation Andrea Scarpino Andrea Scarpino
HADOOP-16080 hadoop-aws does not work with hadoop-client-api Major fs/s3 Keith Turner Chao Sun
HDFS-15660 StorageTypeProto is not compatiable between 3.x and 2.6 Major . Ryan Wu Ryan Wu
HDFS-15707 NNTop counts don’t add up as expected Major hdfs, metrics, namenode Ahmed Hussein Ahmed Hussein
HDFS-15709 EC: Socket file descriptor leak in StripedBlockChecksumReconstructor Major datanode, ec, erasure-coding Yushi Hayasaka Yushi Hayasaka
YARN-10495 make the rpath of container-executor configurable Major yarn angerszhu angerszhu
HDFS-15240 Erasure Coding: dirty buffer causes reconstruction block error Blocker datanode, erasure-coding HuangTao HuangTao
YARN-10491 Fix deprecation warnings in SLSWebApp.java Minor build Akira Ajisaka Ankit Kumar
HADOOP-13571 ServerSocketUtil.getPort() should use loopback address, not 0.0.0.0 Major . Eric Badger Eric Badger
HDFS-15725 Lease Recovery never completes for a committed block which the DNs never finalize Major namenode Stephen O’Donnell Stephen O’Donnell
HDFS-15170 EC: Block gets marked as CORRUPT in case of failover and pipeline recovery Critical erasure-coding Ayush Saxena Ayush Saxena
YARN-10536 Client in distributedShell swallows interrupt exceptions Major client, distributed-shell Ahmed Hussein Ahmed Hussein
HDFS-15743 Fix -Pdist build failure of hadoop-hdfs-native-client Major . Masatake Iwasaki Masatake Iwasaki
YARN-10334 TestDistributedShell leaks resources on timeout/failure Major distributed-shell, test, yarn Ahmed Hussein Ahmed Hussein
YARN-10558 Fix failure of TestDistributedShell#testDSShellWithOpportunisticContainers Minor test Masatake Iwasaki Masatake Iwasaki
HDFS-15719 [Hadoop 3] Both NameNodes can crash simultaneously due to the short JN socket timeout Critical . Wei-Chiu Chuang Wei-Chiu Chuang
YARN-10560 Upgrade node.js to 10.23.1 and yarn to 1.22.5 in Web UI v2 Major webapp, yarn-ui-v2 Akira Ajisaka Akira Ajisaka
HADOOP-17444 ADLFS: Update SDK version from 2.3.6 to 2.3.9 Minor fs/adl Bilahari T H Bilahari T H
YARN-10528 maxAMShare should only be accepted for leaf queues, not parent queues Major . Siddharth Ahuja Siddharth Ahuja
HADOOP-17438 Increase docker memory limit in Jenkins Major build, scripts, test, yetus Ahmed Hussein Ahmed Hussein
MAPREDUCE-7310 Clear the fileMap in JHEventHandlerForSigtermTest Minor test Zhengxi Li Zhengxi Li
HADOOP-16947 Stale record should be remove when MutableRollingAverages generating aggregate data. Major . Haibin Huang Haibin Huang
YARN-10515 Fix flaky test TestCapacitySchedulerAutoQueueCreation.testDynamicAutoQueueCreationWithTags Major test Peter Bacsko Peter Bacsko
HADOOP-17224 Install Intel ISA-L library in Dockerfile Blocker . Takanobu Asanuma Takanobu Asanuma
HDFS-15632 AbstractContractDeleteTest should set recursive parameter to true for recursive test cases. Major . Konstantin Shvachko Anton Kutuzov
HADOOP-17258 MagicS3GuardCommitter fails with `pendingset` already exists Major fs/s3 Dongjoon Hyun Dongjoon Hyun
HDFS-15661 The DeadNodeDetector shouldn’t be shared by different DFSClients. Major . Jinglun Jinglun
HDFS-10498 Intermittent test failure org.apache.hadoop.hdfs.server.namenode.snapshot.TestSnapshotFileLength.testSnapshotfileLength Major hdfs, snapshots Hanisha Koneru Jim Brennan
HADOOP-17506 Fix typo in BUILDING.txt Trivial documentation Gautham Banasandra Gautham Banasandra
HDFS-15791 Possible Resource Leak in FSImageFormatProtobuf Major namenode Narges Shadab Narges Shadab
HDFS-15795 EC: Wrong checksum when reconstruction was failed by exception Major datanode, ec, erasure-coding Yushi Hayasaka Yushi Hayasaka
HDFS-15779 EC: fix NPE caused by StripedWriter.clearBuffers during reconstruct block Major . Hongbing Wang Hongbing Wang
HADOOP-17217 S3A FileSystem does not correctly delete directories with fake entries Major fs/s3 Kaya Kupferschmidt
HDFS-15798 EC: Reconstruct task failed, and It would be XmitsInProgress of DN has negative number Major . Haiyang Hu Haiyang Hu
YARN-10607 User environment is unable to prepend PATH when mapreduce.admin.user.env also sets PATH Major . Eric Badger Eric Badger
HDFS-15792 ClasscastException while loading FSImage Major nn Renukaprasad C Renukaprasad C
HADOOP-17516 Upgrade ant to 1.10.9 Major . Akira Ajisaka Akira Ajisaka
YARN-10500 TestDelegationTokenRenewer fails intermittently Major test Akira Ajisaka Masatake Iwasaki
HDFS-15806 DeadNodeDetector should close all the threads when it is closed. Major . Jinglun Jinglun
HADOOP-17534 Upgrade Jackson databind to 2.10.5.1 Major build Adam Roberts Akira Ajisaka
MAPREDUCE-7323 Remove job_history_summary.py Major . Akira Ajisaka Akira Ajisaka
YARN-10647 Fix TestRMNodeLabelsManager failed after YARN-10501. Major . Qi Zhu Qi Zhu
HADOOP-17528 Not closing an SFTP File System instance prevents JVM from exiting. Major . Mikhail Pryakhin Mikhail Pryakhin
HADOOP-17510 Hadoop prints sensitive Cookie information. Major . Renukaprasad C Renukaprasad C
HDFS-15422 Reported IBR is partially replaced with stored info when queuing. Critical namenode Kihwal Lee Stephen O’Donnell
YARN-10651 CapacityScheduler crashed with NPE in AbstractYarnScheduler.updateNodeResource() Major . Haibo Chen Haibo Chen
MAPREDUCE-7320 ClusterMapReduceTestCase does not clean directories Major . Ahmed Hussein Ahmed Hussein
HDFS-14013 Skip any credentials stored in HDFS when starting ZKFC Major hdfs Krzysztof Adamski Stephen O’Donnell
HDFS-15849 ExpiredHeartbeats metric should be of Type.COUNTER Major metrics Konstantin Shvachko Qi Zhu
YARN-10649 Fix RMNodeImpl.updateExistContainers leak Major resourcemanager Max Xie Max Xie
YARN-10672 All testcases in TestReservations are flaky Major . Szilard Nemeth Szilard Nemeth
HADOOP-17557 skip-dir option is not processed by Yetus Major build, precommit, yetus Ahmed Hussein Ahmed Hussein
YARN-10671 Fix Typo in TestSchedulingRequestContainerAllocation Minor . D M Murali Krishna Reddy D M Murali Krishna Reddy
HDFS-15875 Check whether file is being truncated before truncate Major . Hui Fei Hui Fei
HADOOP-17582 Replace GitHub App Token with GitHub OAuth token Major build Akira Ajisaka Akira Ajisaka
YARN-10687 Add option to disable/enable free disk space checking and percentage checking for full and not-full disks Major nodemanager Qi Zhu Qi Zhu
HADOOP-17586 Upgrade org.codehaus.woodstox:stax2-api to 4.2.1 Major . Ayush Saxena Ayush Saxena
HADOOP-17585 Correct timestamp format in the docs for the touch command Major . Stephen O’Donnell Stephen O’Donnell
HDFS-15809 DeadNodeDetector doesn’t remove live nodes from dead node set. Major . Jinglun Jinglun
HADOOP-17532 Yarn Job execution get failed when LZ4 Compression Codec is used Major common Bhavik Patel Bhavik Patel
YARN-10588 Percentage of queue and cluster is zero in WebUI Major . Bilwa S T Bilwa S T
MAPREDUCE-7322 revisiting TestMRIntermediateDataEncryption Major job submission, security, test Ahmed Hussein Ahmed Hussein
YARN-10703 Fix potential null pointer error of gpuNodeResourceUpdateHandler in NodeResourceMonitorImpl. Major . Qi Zhu Qi Zhu
HDFS-15868 Possible Resource Leak in EditLogFileOutputStream Major . Narges Shadab Narges Shadab
HADOOP-17592 Fix the wrong CIDR range example in Proxy User documentation Minor documentation Kwangsun Noh Kwangsun Noh
YARN-10706 Upgrade com.github.eirslett:frontend-maven-plugin to 1.11.2 Major buid Mingliang Liu Mingliang Liu
MAPREDUCE-7325 Intermediate data encryption is broken in LocalJobRunner Major job submission, security Ahmed Hussein Ahmed Hussein
HDFS-15908 Possible Resource Leak in org.apache.hadoop.hdfs.qjournal.server.Journal Major . Narges Shadab Narges Shadab
HDFS-15910 Replace bzero with explicit_bzero for better safety Critical libhdfs++ Gautham Banasandra Gautham Banasandra
YARN-10697 Resources are displayed in bytes in UI for schedulers other than capacity Major . Bilwa S T Bilwa S T
HADOOP-17602 Upgrade JUnit to 4.13.1 Major build, security, test Ahmed Hussein Ahmed Hussein
HDFS-15900 RBF: empty blockpool id on dfsrouter caused by UNAVAILABLE NameNode Major rbf Harunobu Daikoku Harunobu Daikoku
YARN-10501 Can’t remove all node labels after add node label without nodemanager port Critical yarn caozhiqiang caozhiqiang
YARN-10437 Destroy yarn service if any YarnException occurs during submitApp Minor yarn-native-services D M Murali Krishna Reddy D M Murali Krishna Reddy
YARN-10439 Yarn Service AM listens on all IP’s on the machine Minor security, yarn-native-services D M Murali Krishna Reddy D M Murali Krishna Reddy
YARN-10441 Add support for hadoop.http.rmwebapp.scheduler.page.class Major scheduler D M Murali Krishna Reddy D M Murali Krishna Reddy
YARN-10466 Fix NullPointerException in yarn-services Component.java Minor . D M Murali Krishna Reddy D M Murali Krishna Reddy
YARN-10716 Fix typo in ContainerRuntime Trivial documentation Wanqiang Ji xishuhai
HDFS-15494 TestReplicaCachingGetSpaceUsed#testReplicaCachingGetSpaceUsedByRBWReplica Fails on Windows Major . Ravuri Sushma sree Ravuri Sushma sree
HADOOP-17610 DelegationTokenAuthenticator prints token information Major . Ravuri Sushma sree Ravuri Sushma sree
HADOOP-17587 Kinit with keytab should not display the keytab file’s full path in any logs Major . Ravuri Sushma sree Ravuri Sushma sree
HDFS-15950 Remove unused hdfs.proto import Major hdfs-client Gautham Banasandra Gautham Banasandra
HDFS-15940 Some tests in TestBlockRecovery are consistently failing Major . Viraj Jasani Viraj Jasani
HDFS-15949 Fix integer overflow Major libhdfs++ Gautham Banasandra Gautham Banasandra
HADOOP-17621 hadoop-auth to remove jetty-server dependency Major auth Wei-Chiu Chuang Wei-Chiu Chuang
HDFS-15948 Fix test4tests for libhdfspp Critical build, libhdfs++ Gautham Banasandra Gautham Banasandra
HADOOP-17617 Incorrect representation of RESPONSE for Get Key Version in KMS index.md.vm file Major . Ravuri Sushma sree Ravuri Sushma sree
MAPREDUCE-7329 HadoopPipes task may fail when linux kernel version change from 3.x to 4.x Major pipes chaoli chaoli
HADOOP-17608 Fix TestKMS failure Major kms Akira Ajisaka Akira Ajisaka
HDFS-15963 Unreleased volume references cause an infinite loop Major datanode Shuyan Zhang Shuyan Zhang
YARN-10460 Upgrading to JUnit 4.13 causes tests in TestNodeStatusUpdater to fail Major nodemanager, test Peter Bacsko Peter Bacsko
HADOOP-17641 ITestWasbUriAndConfiguration.testCanonicalServiceName() failing now mockaccount exists Minor fs/azure, test Steve Loughran Steve Loughran
HDFS-15974 RBF: Unable to display the datanode UI of the router Major rbf, ui Xiangyi Zhu Xiangyi Zhu
HADOOP-17655 Upgrade Jetty to 9.4.40 Blocker . Akira Ajisaka Akira Ajisaka
YARN-10749 Can’t remove all node labels after add node label without nodemanager port, broken by YARN-10647 Major . D M Murali Krishna Reddy D M Murali Krishna Reddy
HDFS-15566 NN restart fails after RollingUpgrade from 3.1.3/3.2.1 to 3.3.0 Blocker . Brahma Reddy Battula Brahma Reddy Battula
HDFS-15621 Datanode DirectoryScanner uses excessive memory Major datanode Stephen O’Donnell Stephen O’Donnell
YARN-10752 Shaded guava not found when compiling with profile hbase2.0 Blocker timelineserver Wei-Chiu Chuang Wei-Chiu Chuang
HDFS-15865 Interrupt DataStreamer thread Minor datanode Karthik Palanisamy
HDFS-15810 RBF: RBFMetrics’s TotalCapacity out of bounds Major . Xiaoxing Wei Fengnan Li
HADOOP-17657 SequeneFile.Writer should implement StreamCapabilities Major . Kishen Das Kishen Das
YARN-10756 Remove additional junit 4.11 dependency from javadoc Major build, test, timelineservice ANANDA G B Akira Ajisaka
HADOOP-17375 Fix the error of TestDynamometerInfra Major test Akira Ajisaka Takanobu Asanuma
HDFS-16001 TestOfflineEditsViewer.testStored() fails reading negative value of FSEditLogOpCodes Blocker hdfs Konstantin Shvachko Akira Ajisaka
HADOOP-17142 Fix outdated properties of journal node when perform rollback Minor . Deegue
HADOOP-17107 hadoop-azure parallel tests not working on recent JDKs Major build, fs/azure Steve Loughran Steve Loughran
YARN-10555 Missing access check before getAppAttempts Critical webapp lujie lujie
HADOOP-17703 checkcompatibility.py errors out when specifying annotations Major . Wei-Chiu Chuang Wei-Chiu Chuang
HDFS-16027 HDFS-15245 breaks source code compatibility between 3.3.0 and 3.3.1. Blocker journal-node, ui Wei-Chiu Chuang Wei-Chiu Chuang
YARN-10725 Backport YARN-10120 to branch-3.3 Major . Bilwa S T Bilwa S T
YARN-10701 The yarn.resource-types should support multi types without trimmed. Major . Qi Zhu Qi Zhu
HADOOP-17718 Explicitly set locale in the Dockerfile Blocker build Wei-Chiu Chuang Wei-Chiu Chuang
MAPREDUCE-7348 TestFrameworkUploader#testNativeIO fails Major test Akira Ajisaka Akira Ajisaka
HADOOP-17723 [build] fix the Dockerfile for ARM Blocker build Wei-Chiu Chuang Wei-Chiu Chuang

TESTS:

JIRA Summary Priority Component Reporter Contributor
MAPREDUCE-7288 Fix TestLongLong#testRightShift Minor . Wanqiang Ji Wanqiang Ji
HDFS-15514 Remove useless dfs.webhdfs.enabled Minor test Hui Fei Hui Fei
HADOOP-17205 Move personality file from Yetus to Hadoop repository Major test, yetus Chao Sun Chao Sun
HDFS-15564 Add Test annotation for TestPersistBlocks#testRestartDfsWithSync Minor hdfs Hui Fei Hui Fei
YARN-9333 TestFairSchedulerPreemption.testRelaxLocalityPreemptionWithNoLessAMInRemainingNodes fails intermittently Major yarn Prabhu Joseph Peter Bacsko
HDFS-15690 Add lz4-java as hadoop-hdfs test dependency Major . L. C. Hsieh L. C. Hsieh
HADOOP-17459 ADL Gen1: Fix the test case failures which are failing after the contract test update in hadoop-common Minor fs/adl Bilahari T H Bilahari T H
HDFS-15898 Test case TestOfflineImageViewer fails Minor . Hui Fei Hui Fei

SUB-TASKS:

JIRA Summary Priority Component Reporter Contributor
HADOOP-16857 ABFS: Optimize HttpRequest retry triggers Major fs/azure Sneha Vijayarajan Sneha Vijayarajan
HADOOP-17002 ABFS: Avoid storage calls to check if the account is HNS enabled or not Minor fs/azure Bilahari T H Bilahari T H
YARN-10215 Endpoint for obtaining direct URL for the logs Major yarn Adam Antal Andras Gyori
HDFS-14353 Erasure Coding: metrics xmitsInProgress become to negative. Major datanode, erasure-coding Baolong Mao Baolong Mao
HDFS-15305 Extend ViewFS and provide ViewFSOverloadScheme implementation with scheme configurable. Major fs, hadoop-client, hdfs-client, viewfs Uma Maheswara Rao G Uma Maheswara Rao G
YARN-10259 Reserved Containers not allocated from available space of other nodes in CandidateNodeSet in MultiNodePlacement Major capacityscheduler Prabhu Joseph Prabhu Joseph
HDFS-15306 Make mount-table to read from central place ( Let’s say from HDFS) Major configuration, hadoop-client Uma Maheswara Rao G Uma Maheswara Rao G
HADOOP-16756 distcp -update to S3A; abfs, etc always overwrites due to block size mismatch Major fs/s3, tools/distcp Daisuke Kobayashi Steve Loughran
HDFS-15322 Make NflyFS to work when ViewFsOverloadScheme’s scheme and target uris schemes are same. Major fs, nflyFs, viewfs, viewfsOverloadScheme Uma Maheswara Rao G Uma Maheswara Rao G
YARN-10108 FS-CS converter: nestedUserQueue with default rule results in invalid queue mapping Major . Prabhu Joseph Gergely Pollák
HADOOP-16852 ABFS: Send error back to client for Read Ahead request failure Major fs/azure Sneha Vijayarajan Sneha Vijayarajan
HADOOP-17053 ABFS: FS initialize fails for incompatible account-agnostic Token Provider setting Major fs/azure Sneha Vijayarajan Sneha Vijayarajan
HDFS-15321 Make DFSAdmin tool to work with ViewFSOverloadScheme Major dfsadmin, fs, viewfs Uma Maheswara Rao G Uma Maheswara Rao G
HADOOP-16568 S3A FullCredentialsTokenBinding fails if local credentials are unset Major fs/s3 Steve Loughran Steve Loughran
YARN-10284 Add lazy initialization of LogAggregationFileControllerFactory in LogServlet Major log-aggregation, yarn Adam Antal Adam Antal
HDFS-15330 Document the ViewFSOverloadScheme details in ViewFS guide Major viewfs, viewfsOverloadScheme Uma Maheswara Rao G Uma Maheswara Rao G
HDFS-15389 DFSAdmin should close filesystem and dfsadmin -setBalancerBandwidth should work with ViewFSOverloadScheme Major dfsadmin, viewfsOverloadScheme Ayush Saxena Ayush Saxena
HDFS-15394 Add all available fs.viewfs.overload.scheme.target.<scheme>.impl classes in core-default.xml bydefault. Major configuration, viewfs, viewfsOverloadScheme Uma Maheswara Rao G Uma Maheswara Rao G
HDFS-15387 FSUsage$DF should consider ViewFSOverloadScheme in processPath Minor viewfs Uma Maheswara Rao G Uma Maheswara Rao G
YARN-10292 FS-CS converter: add an option to enable asynchronous scheduling in CapacityScheduler Major fairscheduler Benjamin Teke Benjamin Teke
HADOOP-17004 ABFS: Improve the ABFS driver documentation Minor fs/azure Bilahari T H Bilahari T H
HDFS-15418 ViewFileSystemOverloadScheme should represent mount links as non symlinks Major . Uma Maheswara Rao G Uma Maheswara Rao G
YARN-9930 Support max running app logic for CapacityScheduler Major capacity scheduler, capacityscheduler zhoukang Peter Bacsko
HDFS-15427 Merged ListStatus with Fallback target filesystem and InternalDirViewFS. Major viewfs Uma Maheswara Rao G Uma Maheswara Rao G
YARN-10316 FS-CS converter: convert maxAppsDefault, maxRunningApps settings Major . Peter Bacsko Peter Bacsko
HADOOP-17054 ABFS: Fix idempotency test failures when SharedKey is set as AuthType Major fs/azure Sneha Vijayarajan Sneha Vijayarajan
HADOOP-17050 S3A to support additional token issuers Minor fs/s3 Gabor Bota Steve Loughran
HADOOP-17015 ABFS: Make PUT and POST operations idempotent Major fs/azure Sneha Vijayarajan Sneha Vijayarajan
HDFS-15429 mkdirs should work when parent dir is internalDir and fallback configured. Major . Uma Maheswara Rao G Uma Maheswara Rao G
HDFS-15436 Default mount table name used by ViewFileSystem should be configurable Major viewfs, viewfsOverloadScheme Virajith Jalaparti Virajith Jalaparti
HADOOP-16798 job commit failure in S3A MR magic committer test Major fs/s3 Steve Loughran Steve Loughran
YARN-10325 Document max-parallel-apps for Capacity Scheduler Major capacity scheduler, capacityscheduler Peter Bacsko Peter Bacsko
HADOOP-16961 ABFS: Adding metrics to AbfsInputStream (AbfsInputStreamStatistics) Major fs/azure Gabor Bota Mehakmeet Singh
HADOOP-17086 ABFS: Fix the parsing errors in ABFS Driver with creation Time (being returned in ListPath) Major fs/azure Ishani Bilahari T H
HDFS-15430 create should work when parent dir is internalDir and fallback configured. Major . Uma Maheswara Rao G Uma Maheswara Rao G
HDFS-15450 Fix NN trash emptier to work if ViewFSOveroadScheme enabled Major namenode, viewfsOverloadScheme Uma Maheswara Rao G Uma Maheswara Rao G
HADOOP-17111 Replace Guava Optional with Java8+ Optional Major . Ahmed Hussein Ahmed Hussein
HDFS-15449 Optionally ignore port number in mount-table name when picking from initialized uri Major . Uma Maheswara Rao G Uma Maheswara Rao G
HADOOP-17058 Support for Appendblob in abfs driver Major fs/azure Ishani Ishani
HDFS-15462 Add fs.viewfs.overload.scheme.target.ofs.impl to core-default.xml Major configuration, viewfs, viewfsOverloadScheme Siyao Meng Siyao Meng
HDFS-15464 ViewFsOverloadScheme should work when -fs option pointing to remote cluster without mount links Major viewfsOverloadScheme Uma Maheswara Rao G Uma Maheswara Rao G
HADOOP-17105 S3AFS globStatus attempts to resolve symlinks Minor fs/s3 Jimmy Zuber Jimmy Zuber
HADOOP-17022 Tune S3A listFiles() api. Major fs/s3 Mukund Thakur Mukund Thakur
HADOOP-17101 Replace Guava Function with Java8+ Function Major . Ahmed Hussein Ahmed Hussein
HADOOP-17099 Replace Guava Predicate with Java8+ Predicate Minor . Ahmed Hussein Ahmed Hussein
HADOOP-16682 Remove unnecessary ABFS toString() invocations Minor fs/azure Jeetesh Mangwani Bilahari T H
HADOOP-17136 ITestS3ADirectoryPerformance.testListOperations failing Minor fs/s3, test Mukund Thakur Mukund Thakur
HDFS-15478 When Empty mount points, we are assigning fallback link to self. But it should not use full URI for target fs. Major . Uma Maheswara Rao G Uma Maheswara Rao G
HADOOP-17100 Replace Guava Supplier with Java8+ Supplier in Hadoop Major . Ahmed Hussein Ahmed Hussein
HADOOP-17132 ABFS: Fix For Idempotency code Major fs/azure Sneha Vijayarajan Sneha Vijayarajan
HADOOP-17092 ABFS: Long waits and unintended retries when multiple threads try to fetch token using ClientCreds Major fs/azure Sneha Vijayarajan Bilahari T H
HADOOP-17131 Refactor S3A Listing code for better isolation Major fs/s3 Mukund Thakur Mukund Thakur
HADOOP-17137 ABFS: Tests ITestAbfsNetworkStatistics need to be config setting agnostic Minor fs/azure, test Sneha Vijayarajan Bilahari T H
HADOOP-17149 ABFS: Test failure: testFailedRequestWhenCredentialsNotCorrect fails when run with SharedKey Minor fs/azure Sneha Vijayarajan Bilahari T H
HADOOP-17163 ABFS: Add debug log for rename failures Major fs/azure Bilahari T H Bilahari T H
HDFS-15515 mkdirs on fallback should throw IOE out instead of suppressing and returning false Major . Uma Maheswara Rao G Uma Maheswara Rao G
HADOOP-13230 S3A to optionally retain directory markers Major fs/s3 Aaron Fabbri Steve Loughran
HADOOP-14124 S3AFileSystem silently deletes “fake” directories when writing a file. Minor fs, fs/s3 Joel Baranick
HADOOP-16966 ABFS: Upgrade Store REST API Version to 2019-12-12 Major fs/azure Ishani Sneha Vijayarajan
HDFS-15533 Provide DFS API compatible class(ViewDistributedFileSystem), but use ViewFileSystemOverloadScheme inside Major dfs, viewfs Uma Maheswara Rao G Uma Maheswara Rao G
HADOOP-17074 Optimise s3a Listing to be fully asynchronous. Major fs/s3 Mukund Thakur Mukund Thakur
HDFS-15529 getChildFilesystems should include fallback fs as well Critical viewfs, viewfsOverloadScheme Uma Maheswara Rao G Uma Maheswara Rao G
HADOOP-17167 ITestS3AEncryptionWithDefaultS3Settings fails if default bucket encryption != KMS Minor fs/s3 Steve Loughran Mukund Thakur
HADOOP-17227 improve s3guard markers command line tool Major fs/s3 Steve Loughran Steve Loughran
YARN-10332 RESOURCE_UPDATE event was repeatedly registered in DECOMMISSIONING state Minor resourcemanager yehuanhuan yehuanhuan
HDFS-15558 ViewDistributedFileSystem#recoverLease should call super.recoverLease when there are no mounts configured Major . Uma Maheswara Rao G Uma Maheswara Rao G
HADOOP-17181 Handle transient stream read failures in FileSystem contract tests Minor fs/s3 Steve Loughran
HDFS-15551 Tiny Improve for DeadNode detector Minor hdfs-client dark_num imbajin
HDFS-15555 RBF: Refresh cacheNS when SocketException occurs Major rbf Akira Ajisaka Akira Ajisaka
HDFS-15532 listFiles on root/InternalDir will fail if fallback root has file Major . Uma Maheswara Rao G Uma Maheswara Rao G
HDFS-15578 Fix the rename issues with fallback fs enabled Major viewfs, viewfsOverloadScheme Uma Maheswara Rao G Uma Maheswara Rao G
HDFS-15585 ViewDFS#getDelegationToken should not throw UnsupportedOperationException. Major . Uma Maheswara Rao G Uma Maheswara Rao G
HADOOP-17215 ABFS: Support for conditional overwrite Major fs/azure Sneha Vijayarajan Sneha Vijayarajan
HDFS-14811 RBF: TestRouterRpc#testErasureCoding is flaky Major rbf Chen Zhang Chen Zhang
HADOOP-17023 Tune listStatus() api of s3a. Major fs/s3 Mukund Thakur Mukund Thakur
HADOOP-17279 ABFS: Test testNegativeScenariosForCreateOverwriteDisabled fails for non-HNS account Major fs/azure Sneha Vijayarajan Sneha Vijayarajan
HADOOP-17183 ABFS: Enable checkaccess API Major fs/azure Bilahari T H Bilahari T H
HDFS-15613 RBF: Router FSCK fails after HDFS-14442 Major rbf Akira Ajisaka Akira Ajisaka
HADOOP-17281 Implement FileSystem.listStatusIterator() in S3AFileSystem Major fs/s3 Mukund Thakur Mukund Thakur
HADOOP-17293 S3A to always probe S3 in S3A getFileStatus on non-auth paths Major fs/s3 Steve Loughran Steve Loughran
HDFS-15620 RBF: Fix test failures after HADOOP-17281 Major rbf, test Akira Ajisaka Akira Ajisaka
HADOOP-17261 s3a rename() now requires s3:deleteObjectVersion permission Major fs/s3 Steve Loughran Steve Loughran
HADOOP-16915 ABFS: Test failure ITestAzureBlobFileSystemRandomRead.testRandomReadPerformance Major . Bilahari T H Bilahari T H
HADOOP-17166 ABFS: configure output stream thread pool Minor fs/azure Bilahari T H Bilahari T H
HADOOP-17301 ABFS: read-ahead error reporting breaks buffer management Critical fs/azure Sneha Vijayarajan Sneha Vijayarajan
HADOOP-17288 Use shaded guava from thirdparty Major . Ayush Saxena Ayush Saxena
HDFS-15459 TestBlockTokenWithDFSStriped fails intermittently Major hdfs Ahmed Hussein Ahmed Hussein
HDFS-15461 TestDFSClientRetries#testGetFileChecksum fails intermittently Major dfsclient, test Ahmed Hussein Ahmed Hussein
HDFS-9776 TestHAAppend#testMultipleAppendsDuringCatchupTailing is flaky Major . Vinayakumar B Ahmed Hussein
HDFS-15657 RBF: TestRouter#testNamenodeHeartBeatEnableDefault fails by BindException Major rbf, test Akira Ajisaka Akira Ajisaka
HADOOP-17305 ITestCustomSigner fails with gcs s3 compatible endpoint. Major fs/s3 Mukund Thakur Mukund Thakur
HDFS-15643 EC: Fix checksum computation in case of native encoders Blocker . Ahmed Hussein Ayush Saxena
HADOOP-17344 Harmonize guava version and shade guava in yarn-csi Major . Wei-Chiu Chuang Akira Ajisaka
HADOOP-17376 ITestS3AContractRename failing against stricter tests Major fs/s3, test Steve Loughran Attila Doroszlai
HADOOP-17379 AbstractS3ATokenIdentifier to set issue date == now Major fs/s3 Steve Loughran Jungtaek Lim
HADOOP-17244 HADOOP-17244. S3A directory delete tombstones dir markers prematurely. Blocker fs/s3 Steve Loughran Steve Loughran
HADOOP-17388 AbstractS3ATokenIdentifier to issue date in UTC Major . Steve Loughran Jungtaek Lim
HADOOP-17343 Upgrade aws-java-sdk to 1.11.901 Minor build, fs/s3 Dongjoon Hyun Steve Loughran
HADOOP-17325 WASB: Test failures Major fs/azure, test Sneha Vijayarajan Steve Loughran
HADOOP-17323 s3a getFileStatus(“/”) to skip IO Minor fs/s3 Steve Loughran Mukund Thakur
HADOOP-17311 ABFS: Logs should redact SAS signature Major fs/azure, security Sneha Vijayarajan Bilahari T H
HADOOP-17313 FileSystem.get to support slow-to-instantiate FS clients Major fs, fs/azure, fs/s3 Steve Loughran Steve Loughran
HADOOP-17394 [JDK 11] mvn package -Pdocs fails Major build, documentation Akira Ajisaka Akira Ajisaka
HADOOP-17396 ABFS: testRenameFileOverExistingFile Fails after Contract test update Major fs/azure, test Sneha Vijayarajan Sneha Vijayarajan
HADOOP-17318 S3A committer to support concurrent jobs with same app attempt ID & dest dir Minor fs/s3 Steve Loughran Steve Loughran
HADOOP-17385 ITestS3ADeleteCost.testDirMarkersFileCreation failure Major fs/s3 Steve Loughran Steve Loughran
HADOOP-15844 tag S3GuardTool entry points as limitedPrivate(“management-tools”)/evolving Minor fs/s3 Steve Loughran Steve Loughran
HADOOP-17332 S3A marker tool mixes up -min and -max Trivial fs/s3 Steve Loughran Steve Loughran
HADOOP-17397 ABFS: SAS Test updates for version and permission update Major fs/azure Sneha Vijayarajan Sneha Vijayarajan
HDFS-15708 TestURLConnectionFactory fails by NoClassDefFoundError in branch-3.3 and branch-3.2 Blocker test Akira Ajisaka Chao Sun
HDFS-15716 TestUpgradeDomainBlockPlacementPolicy flaky Major namenode, test Ahmed Hussein Ahmed Hussein
HADOOP-17422 ABFS: Set default ListMaxResults to max server limit Major fs/azure Sumangala Patki Thomas Marqardt
HADOOP-17450 hadoop-common to add IOStatistics API Major fs Steve Loughran Steve Loughran
HADOOP-17271 S3A statistics to support IOStatistics Major fs/s3 Steve Loughran Steve Loughran
HADOOP-17347 ABFS: Optimise read for small files/tails of files Major fs/azure Bilahari T H Bilahari T H
HADOOP-17272 ABFS Streams to support IOStatistics API Major fs/azure Steve Loughran Mehakmeet Singh
HADOOP-17451 Intermittent failure of S3A tests which make assertions on statistics/IOStatistics Major fs/s3 Steve Loughran Steve Loughran
HDFS-15762 TestMultipleNNPortQOP#testMultipleNNPortOverwriteDownStream fails intermittently Minor . Toshihiko Uchida Toshihiko Uchida
HDFS-15672 TestBalancerWithMultipleNameNodes#testBalancingBlockpoolsWithBlockPoolPolicy fails on trunk Major . Ahmed Hussein Masatake Iwasaki
HADOOP-13845 s3a to instrument duration of HTTP calls Minor fs/s3 Steve Loughran Steve Loughran
HADOOP-17456 S3A ITestPartialRenamesDeletes.testPartialDirDelete[bulk-delete=true] failure Minor fs/s3, test Steve Loughran Steve Loughran
HADOOP-17455 [s3a] Intermittent failure of ITestS3ADeleteCost.testDeleteSingleFileInDir Major fs/s3, test Gabor Bota Steve Loughran
HADOOP-17433 Skipping network I/O in S3A getFileStatus(/) breaks ITestAssumeRole Minor fs/s3, test Steve Loughran Steve Loughran
HADOOP-17296 ABFS: Allow Random Reads to be of Buffer Size Major fs/azure Sneha Vijayarajan Sneha Vijayarajan
HADOOP-17413 ABFS: Release Elastic ByteBuffer pool memory at outputStream close Major fs/azure Sneha Vijayarajan Sneha Vijayarajan
HADOOP-17407 ABFS: Delete Idempotency handling can lead to NPE Major fs/azure Sneha Vijayarajan Sneha Vijayarajan
HADOOP-17404 ABFS: Piggyback flush on Append calls for short writes Major fs/azure Sneha Vijayarajan Sneha Vijayarajan
HADOOP-17480 S3A docs to state s3 is consistent, deprecate S3Guard Major fs/s3 Steve Loughran Steve Loughran
HADOOP-17414 Magic committer files don’t have the count of bytes written collected by spark Major fs/s3 Steve Loughran Steve Loughran
HADOOP-17493 renaming S3A Statistic DELEGATION_TOKENS_ISSUED to DELEGATION_TOKEN_ISSUED broke tests downstream Major fs/s3 Steve Loughran Steve Loughran
HADOOP-17483 magic committer to be enabled for all S3 buckets Major fs/s3 Steve Loughran Steve Loughran
HADOOP-17337 S3A NetworkBinding has a runtime class dependency on a third-party shaded class Blocker fs/s3 Chris Wensel Steve Loughran
HADOOP-17475 ABFS : add high performance listStatusIterator Major fs/azure Bilahari T H Bilahari T H
HADOOP-17432 [JDK 16] KerberosUtil#getOidInstance is broken by JEP 396 Major auth Akira Ajisaka Akira Ajisaka
HADOOP-13327 Add OutputStream + Syncable to the Filesystem Specification Major fs Steve Loughran Steve Loughran
HDFS-15836 RBF: Fix contract tests after HADOOP-13327 Major rbf Akira Ajisaka Akira Ajisaka
HADOOP-17038 Support disabling buffered reads in ABFS positional reads Major . Anoop Sam John Anoop Sam John
HADOOP-15710 ABFS checkException to map 403 to AccessDeniedException Blocker fs/azure Steve Loughran Steve Loughran
HDFS-15847 create client protocol: add ecPolicyName & storagePolicy param to debug statement string Minor . Bhavik Patel Bhavik Patel
HADOOP-16748 Migrate to Python 3 and upgrade Yetus to 0.13.0 Major . Akira Ajisaka Akira Ajisaka
HADOOP-16906 Add some Abortable.abort() interface for streams etc which can be terminated Blocker fs, fs/azure, fs/s3 Steve Loughran Jungtaek Lim
HADOOP-17567 typo in MagicCommitTracker Trivial fs/s3 Pierrick HYMBERT Pierrick HYMBERT
HADOOP-17191 ABFS: Run the integration tests with various combinations of configurations and publish consolidated results Minor fs/azure, test Bilahari T H Bilahari T H
HADOOP-16721 Improve S3A rename resilience Blocker fs/s3 Steve Loughran Steve Loughran
HADOOP-17548 ABFS: Toggle Store Mkdirs request overwrite parameter Major fs/azure Sumangala Patki Sumangala Patki
HADOOP-17537 Correct abfs test assertion reversed in HADOOP-13327 Major fs/azure, test Sumangala Patki Sumangala Patki
HDFS-15890 Improve the Logs for File Concat Operation Minor namenode Bhavik Patel Bhavik Patel
HDFS-13975 TestBalancer#testMaxIterationTime fails sporadically Major . Jason Darrell Lowe Toshihiko Uchida
YARN-10688 ClusterMetrics should support GPU capacity related metrics. Major metrics, resourcemanager Qi Zhu Qi Zhu
YARN-10692 Add Node GPU Utilization and apply to NodeMetrics. Major . Qi Zhu Qi Zhu
HDFS-15902 Improve the log for HTTPFS server operation Minor httpfs Bhavik Patel Bhavik Patel
HADOOP-17476 ITestAssumeRole.testAssumeRoleBadInnerAuth failure Major fs/s3, test Steve Loughran Steve Loughran
HADOOP-13551 Collect AwsSdkMetrics in S3A FileSystem IOStatistics Blocker fs/s3 Steve Loughran Steve Loughran
YARN-10713 ClusterMetrics should support custom resource capacity related metrics. Major . Qi Zhu Qi Zhu
HDFS-15921 Improve the log for the Storage Policy Operations Minor namenode Bhavik Patel Bhavik Patel
YARN-10702 Add cluster metric for amount of CPU used by RM Event Processor Minor yarn Jim Brennan Jim Brennan
YARN-10503 Support queue capacity in terms of absolute resources with custom resourceType. Critical . Qi Zhu Qi Zhu
HADOOP-17630 [JDK 15] TestPrintableString fails due to Unicode 13.0 support Major . Akira Ajisaka Akira Ajisaka
HADOOP-17576 ABFS: Disable throttling update for auth failures Major fs/azure Sumangala Patki Sumangala Patki
YARN-10723 Change CS nodes page in UI to support custom resource. Major . Qi Zhu Qi Zhu
HADOOP-16948 ABFS: Support infinite lease dirs Minor . Billie Rinaldi Billie Rinaldi
HADOOP-17471 ABFS to collect IOStatistics Major fs/azure Steve Loughran Mehakmeet Singh
HADOOP-17535 ABFS: ITestAzureBlobFileSystemCheckAccess test failure if test doesn’t have oauth keys Major fs/azure Steve Loughran Steve Loughran
HADOOP-17112 whitespace not allowed in paths when saving files to s3a via committer Blocker fs/s3 Krzysztof Adamski Krzysztof Adamski
HADOOP-17597 Add option to downgrade S3A rejection of Syncable to warning Minor . Steve Loughran Steve Loughran
HADOOP-17661 mvn versions:set fails to parse pom.xml Blocker build Wei-Chiu Chuang Wei-Chiu Chuang
HADOOP-17536 ABFS: Suport for customer provided encryption key Minor fs/azure Bilahari T H Bilahari T H
YARN-10707 Support custom resources in ResourceUtilization, and update Node GPU Utilization to use. Major yarn Qi Zhu Qi Zhu
HADOOP-17653 Do not use guava’s Files.createTempDir() Major . Wei-Chiu Chuang Wei-Chiu Chuang
HDFS-15952 TestRouterRpcMultiDestination#testProxyGetTransactionID and testProxyVersionRequest are flaky Major rbf Harunobu Daikoku Akira Ajisaka
HADOOP-16742 Possible NPE in S3A MultiObjectDeleteSupport error handling Major fs/s3 Steve Loughran Steve Loughran
HADOOP-17644 Add back the exceptions removed by HADOOP-17432 for compatibility Blocker bulid Akira Ajisaka Quan Li
YARN-10642 Race condition: AsyncDispatcher can get stuck by the changes introduced in YARN-8995 Critical resourcemanager zhengchenyu zhengchenyu
YARN-9615 Add dispatcher metrics to RM Major . Jonathan Hung Qi Zhu
HDFS-13934 Multipart uploaders to be created through API call to FileSystem/FileContext, not service loader Major fs, fs/s3, hdfs Steve Loughran Steve Loughran
HADOOP-17665 Ignore missing keystore configuration in reloading mechanism Major . Borislav Iordanov Borislav Iordanov
HADOOP-17663 Remove useless property hadoop.assemblies.version in pom file Trivial build Wei-Chiu Chuang Akira Ajisaka
HADOOP-17666 Update LICENSE for 3.3.1 Blocker . Wei-Chiu Chuang Wei-Chiu Chuang

OTHER:

JIRA Summary Priority Component Reporter Contributor
HADOOP-17430 Restore ability to set Text to empty byte array Minor common gaozhan ding gaozhan ding
HDFS-15870 Remove unused configuration dfs.namenode.stripe.min Minor . tomscut tomscut
HDFS-15808 Add metrics for FSNamesystem read/write lock hold long time Major hdfs tomscut tomscut
HDFS-15873 Add namenode address in logs for block report Minor datanode, hdfs tomscut tomscut
HDFS-15906 Close FSImage and FSNamesystem after formatting is complete Minor . tomscut tomscut
HDFS-15892 Add metric for editPendingQ in FSEditLogAsync Minor . tomscut tomscut
HDFS-15951 Remove unused parameters in NameNodeProxiesClient Minor . tomscut tomscut
HDFS-15975 Use LongAdder instead of AtomicLong Minor . tomscut tomscut
HDFS-15970 Print network topology on the web Minor . tomscut tomscut
HDFS-15991 Add location into datanode info for NameNodeMXBean Minor . tomscut tomscut
HADOOP-17055 Remove residual code of Ozone Major . Wanqiang Ji Wanqiang Ji
YARN-10274 Merge QueueMapping and QueueMappingEntity Major yarn Gergely Pollák Gergely Pollák
YARN-10281 Redundant QueuePath usage in UserGroupMappingPlacementRule and AppNameMappingPlacementRule Major . Gergely Pollák Gergely Pollák
YARN-10279 Avoid unnecessary QueueMappingEntity creations Minor . Gergely Pollák Hudáky Márton Gyula
YARN-10277 CapacityScheduler test TestUserGroupMappingPlacementRule should build proper hierarchy Major . Gergely Pollák Szilard Nemeth
HADOOP-16990 Update Mockserver Major . Wei-Chiu Chuang Attila Doroszlai
YARN-10278 CapacityScheduler test framework ProportionalCapacityPreemptionPolicyMockFramework need some review Major . Gergely Pollák Szilard Nemeth
YARN-10540 Node page is broken in YARN UI1 and UI2 including RMWebService api for nodes Critical webapp Sunil G Jim Brennan
HADOOP-17445 Update the year to 2021 Major . Xiaoqiao He Xiaoqiao He
HDFS-15731 Reduce threadCount for unit tests to reduce the memory usage Major build, test Akira Ajisaka Akira Ajisaka
HADOOP-17571 Upgrade com.fasterxml.woodstox:woodstox-core for security reasons Major . Viraj Jasani Viraj Jasani
HDFS-15895 DFSAdmin#printOpenFiles has redundant String#format usage Minor . Viraj Jasani Viraj Jasani
HDFS-15926 Removed duplicate dependency of hadoop-annotations Minor . Viraj Jasani Viraj Jasani
HADOOP-17614 Bump netty to the latest 4.1.61 Blocker . Wei-Chiu Chuang Wei-Chiu Chuang
HADOOP-17622 Avoid usage of deprecated IOUtils#cleanup API Minor . Viraj Jasani Viraj Jasani
HADOOP-17624 Remove any rocksdb exclusion code Major . Wei-Chiu Chuang Wei-Chiu Chuang
HADOOP-17625 Update to Jetty 9.4.39 Major . Wei-Chiu Chuang Wei-Chiu Chuang
HDFS-15989 Split TestBalancer into two classes Major . Viraj Jasani Viraj Jasani
HADOOP-17676 Restrict imports from org.apache.curator.shaded Major . Viraj Jasani Viraj Jasani
HADOOP-17683 Update commons-io to 2.8.0 Major . Wei-Chiu Chuang Akira Ajisaka
HADOOP-17426 Upgrade to hadoop-thirdparty-1.1.0 Major . Ayush Saxena Wei-Chiu Chuang
HADOOP-17739 Use hadoop-thirdparty 1.1.1 Major . Wei-Chiu Chuang Wei-Chiu Chuang