Apache Hadoop Changelog

Release 3.2.1 - 2019-09-10

INCOMPATIBLE CHANGES:

JIRA Summary Priority Component Reporter Contributor
HADOOP-15922 DelegationTokenAuthenticationFilter get wrong doAsUser since it does not decode URL Major common, kms He Xiaoqiao He Xiaoqiao

NEW FEATURES:

JIRA Summary Priority Component Reporter Contributor
HADOOP-15950 Failover for LdapGroupsMapping Major common, security Lukas Majercak Lukas Majercak
YARN-7055 YARN Timeline Service v.2: beta 1 / GA Major timelineclient, timelinereader, timelineserver Vrushali C
YARN-9761 Allow overriding application submissions based on server side configs Major . Jonathan Hung pralabhkumar

IMPROVEMENTS:

JIRA Summary Priority Component Reporter Contributor
HADOOP-15676 Cleanup TestSSLHttpServer Minor common Szilard Nemeth Szilard Nemeth
YARN-8896 Limit the maximum number of container assignments per heartbeat Major . Weiwei Yang Zhankun Tang
YARN-8618 Yarn Service: When all the components of a service have restart policy NEVER then initiation of service upgrade should fail Major . Chandni Singh Chandni Singh
HADOOP-15804 upgrade to commons-compress 1.18 Major . PJ Fanning Akira Ajisaka
YARN-8916 Define a constant “docker” string in “ContainerRuntimeConstants.java” for better maintainability Minor . Zhankun Tang Zhankun Tang
YARN-8908 Fix errors in yarn-default.xml related to GPU/FPGA Major . Zhankun Tang Zhankun Tang
HDFS-13941 make storageId in BlockPoolTokenSecretManager.checkAccess optional Major . Ajay Kumar Ajay Kumar
HDFS-14029 Sleep in TestLazyPersistFiles should be put into a loop Trivial hdfs Adam Antal Adam Antal
YARN-8915 Update the doc about the default value of “maximum-container-assignments” for capacity scheduler Minor . Zhankun Tang Zhankun Tang
HADOOP-15855 Review hadoop credential doc, including object store details Minor documentation, security Steve Loughran Steve Loughran
YARN-7225 Add queue and partition info to RM audit log Major resourcemanager Jonathan Hung Eric Payne
HADOOP-15687 Credentials class should allow access to aliases Trivial . Lars Francke Lars Francke
YARN-8969 AbstractYarnScheduler#getNodeTracker should return generic type to avoid type casting Major . Wanqiang Ji Wanqiang Ji
YARN-8977 Remove unnecessary type casting when calling AbstractYarnScheduler#getSchedulerNode Trivial . Wanqiang Ji Wanqiang Ji
HDFS-14070 Refactor NameNodeWebHdfsMethods to allow better extensibility Major . CR Hota CR Hota
HADOOP-15926 Document upgrading the section in NOTICE.txt when upgrading the version of AWS SDK Minor documentation Akira Ajisaka Dinesh Chitlangia
HADOOP-12558 distcp documentation is woefully out of date Critical documentation, tools/distcp Allen Wittenauer Dinesh Chitlangia
HDFS-14063 Support noredirect param for CREATE/APPEND/OPEN/GETFILECHECKSUM in HttpFS Major . Íñigo Goiri Íñigo Goiri
HADOOP-15919 AliyunOSS: Enable Yarn to use OSS Major fs/oss wujinhu wujinhu
HDFS-14064 WEBHDFS: Support Enable/Disable EC Policy Major . Ayush Saxena Ayush Saxena
HADOOP-15943 AliyunOSS: add missing owner & group attributes for oss FileStatus Major fs/oss wujinhu wujinhu
MAPREDUCE-7164 FileOutputCommitter does not report progress while merging paths. Major . Kuhu Shukla Kuhu Shukla
YARN-9069 Fix SchedulerInfo#getSchedulerType for custom schedulers Minor . Bilwa S T Bilwa S T
HDFS-14095 EC: Track Erasure Coding commands in DFS statistics Major erasure-coding Ayush Saxena Ayush Saxena
HDFS-14112 Avoid recursive call to external authorizer for getContentSummary. Critical namenode Jitendra Nath Pandey Tsz Wo Nicholas Sze
YARN-9036 Escape newlines in health report in YARN UI Major . Jonathan Hung Keqiu Hu
YARN-9041 Performance Optimization of method FSPreemptionThread#identifyContainersToPreempt Major fairscheduler, scheduler preemption Wanqiang Ji Wanqiang Ji
YARN-9085 Add Guaranteed and MaxCapacity to CSQueueMetrics Major . Jonathan Hung Jonathan Hung
HDFS-14124 EC : Support EC Commands (set/get/unset EcPolicy) via WebHdfs Major erasure-coding, httpfs, webhdfs Souryakanta Dwivedy Ayush Saxena
HADOOP-15808 Harden Token service loader use Major security Steve Loughran Steve Loughran
YARN-9122 Add table of contents to YARN Service API document Minor documentation Akira Ajisaka Zhankun Tang
HDFS-14171 Performance improvement in Tailing EditLog Major namenode Kenneth Yang Kenneth Yang
HADOOP-15481 Emit FairCallQueue stats as metrics Major metrics, rpc-server Erik Krogen Christopher Gregorian
HADOOP-15994 Upgrade Jackson2 to 2.9.8 Major security Akira Ajisaka lqjacklee
HADOOP-16019 ZKDelegationTokenSecretManager won’t log exception message occured in function setJaasConfiguration Minor common luhuachao luhuachao
HDFS-14213 Remove Jansson from BUILDING.txt Minor documentation Akira Ajisaka Dinesh Chitlangia
HDFS-14221 Replace Guava Optional with Java Optional Major . Arpit Agarwal Arpit Agarwal
HDFS-14222 Make ThrottledAsyncChecker constructor public Major . Arpit Agarwal Arpit Agarwal
HADOOP-16075 Upgrade checkstyle version to 8.16 Minor build Dinesh Chitlangia Dinesh Chitlangia
HADOOP-16089 AliyunOSS: update oss-sdk version to 3.4.1 Major fs/oss wujinhu wujinhu
HDFS-14231 DataXceiver#run() should not log exceptions caused by InvalidToken exception as an error Major hdfs Kitti Nanasi Kitti Nanasi
YARN-7171 RM UI should sort memory / cores numerically Major . Eric Maynard Ahmed Hussein
YARN-9282 Typo in javadoc of class LinuxContainerExecutor: hadoop.security.authetication should be ‘authentication’ Trivial . Szilard Nemeth Charan Hebri
HADOOP-16108 Tail Follow Interval Should Allow To Specify The Sleep Interval To Save Unnecessary RPC’s Major . Harshakiran Reddy Ayush Saxena
YARN-8295 [UI2] Improve “Resource Usage” tab error message when there are no data available. Minor yarn-ui-v2 Gergely Novák Charan Hebri
YARN-7824 [UI2] Yarn Component Instance page should include link to container logs Major yarn-ui-v2 Yesha Vora Akhil PB
HADOOP-15281 Distcp to add no-rename copy option Major tools/distcp Steve Loughran Andrew Olson
YARN-9309 Improve graph text in SLS to avoid overlapping Minor . Bilwa S T Bilwa S T
HDFS-14235 Handle ArrayIndexOutOfBoundsException in DataNodeDiskMetrics#slowDiskDetectionDaemon Major . Surendra Singh Lilhore Ranith Sardar
YARN-9168 DistributedShell client timeout should be -1 by default Minor . Zhankun Tang Zhankun Tang
YARN-9087 Improve logging for initialization of Resource plugins Major yarn Szilard Nemeth Szilard Nemeth
YARN-9121 Replace GpuDiscoverer.getInstance() to a readable object for easy access control Major . Szilard Nemeth Szilard Nemeth
YARN-9139 Simplify initializer code of GpuDiscoverer Major . Szilard Nemeth Szilard Nemeth
HDFS-14247 Repeat adding node description into network topology Minor datanode HuangTao HuangTao
YARN-9332 RackResolver tool should accept multiple hosts Minor yarn Lantao Jin Lantao Jin
HADOOP-16140 hadoop fs expunge to add -immediate option to purge trash immediately Major fs Stephen O’Donnell Stephen O’Donnell
YARN-9138 Improve test coverage for nvidia-smi binary execution of GpuDiscoverer Major . Szilard Nemeth Szilard Nemeth
MAPREDUCE-7191 JobHistoryServer should log exception when loading/parsing history file failed Minor mrv2 Jiandan Yang Jiandan Yang
MAPREDUCE-7192 JobHistoryServer attempts page support jump to containers log page in NM when logAggregation is disable Major mrv2 Jiandan Yang Jiandan Yang
HDFS-14346 Better time precision in getTimeDuration Minor namenode Chao Sun Chao Sun
HDFS-14366 Improve HDFS append performance Major hdfs Chao Sun Chao Sun
HADOOP-16196 Path Parameterize Comparable Minor common David Mollitor David Mollitor
HADOOP-16181 HadoopExecutors shutdown Cleanup Minor util David Mollitor David Mollitor
HADOOP-16147 Allow CopyListing sequence file keys and values to be more easily customized Major tools/distcp Andrew Olson Andrew Olson
MAPREDUCE-7190 Add SleepJob additional parameter to make parallel runs distinguishable Major . Adam Antal Adam Antal
YARN-9394 Use new API of RackResolver to get better performance Major yarn Lantao Jin Lantao Jin
HADOOP-16208 Do Not Log InterruptedException in Client Minor common David Mollitor David Mollitor
YARN-9463 Add queueName info when failing with queue capacity sanity check Trivial capacity scheduler Aihua Xu Aihua Xu
HADOOP-16227 Upgrade checkstyle to 8.19 Major build Akira Ajisaka Akira Ajisaka
HDFS-14432 dfs.datanode.shared.file.descriptor.paths duplicated in hdfs-default.xml Minor hdfs puleya7 puleya7
HDFS-14463 Add Log Level link under NameNode and DataNode Web UI Utilities dropdown Trivial webhdfs Siyao Meng Siyao Meng
YARN-9529 Log correct cpu controller path on error while initializing CGroups. Major nodemanager Jonathan Hung Jonathan Hung
HADOOP-16289 Allow extra jsvc startup option in hadoop_start_secure_daemon in hadoop-functions.sh Major scripts Siyao Meng Siyao Meng
HADOOP-16307 Intern User Name and Group Name in FileStatus Major fs David Mollitor David Mollitor
HADOOP-16294 Enable access to context by DistCp subclasses Trivial tools/distcp Andrew Olson Andrew Olson
HDFS-14507 Document -blockingDecommission option for hdfs dfsadmin -listOpenFiles Minor documentation Siyao Meng Siyao Meng
HDFS-14451 Incorrect header or version mismatch log message Minor ipc David Mollitor Shweta
HDFS-14502 keepResults option in NNThroughputBenchmark should call saveNamespace() Major benchmarks, hdfs Konstantin Shvachko Konstantin Shvachko
HADOOP-16323 https everywhere in Maven settings Minor build Akira Ajisaka Akira Ajisaka
YARN-9563 Resource report REST API could return NaN or Inf Minor . Ahmed Hussein Ahmed Hussein
YARN-9545 Create healthcheck REST endpoint for ATSv2 Major ATSv2 Zoltan Siegl Zoltan Siegl
HDFS-10659 Namenode crashes after Journalnode re-installation in an HA cluster due to missing paxos directory Major ha, journal-node Amit Anand star
HDFS-14513 FSImage which is saving should be clean while NameNode shutdown Major namenode He Xiaoqiao He Xiaoqiao
YARN-9543 [UI2] Handle ATSv2 server down or failures cases gracefully in YARN UI v2 Major ATSv2, yarn-ui-v2 Zoltan Siegl Zoltan Siegl
HADOOP-16369 Fix zstandard shortname misspelled as zts Major . Jonathan Eagles Jonathan Eagles
HDFS-14560 Allow block replication parameters to be refreshable Major namenode Stephen O’Donnell Stephen O’Donnell
HDFS-12770 Add doc about how to disable client socket cache Trivial hdfs-client Weiwei Yang Weiwei Yang
HADOOP-9157 Better option for curl in hadoop-auth-examples Minor documentation Jingguo Yao Andras Bokor
HDFS-14340 Lower the log level when can’t get postOpAttr Minor nfs Anuhan Torgonshar Anuhan Torgonshar
HADOOP-15914 hadoop jar command has no help argument Major common Adam Antal Adam Antal
YARN-9630 [UI2] Add a link in docs’s top page Major documentation, yarn-ui-v2 Wanqiang Ji Wanqiang Ji
HADOOP-16156 [Clean-up] Remove NULL check before instanceof and fix checkstyle in InnerNodeImpl Minor . Shweta Shweta
HADOOP-14385 HttpExceptionUtils#validateResponse swallows exceptions Trivial . Wei-Chiu Chuang Wei-Chiu Chuang
HDFS-12564 Add the documents of swebhdfs configurations on the client side Major documentation, webhdfs Takanobu Asanuma Takanobu Asanuma
HDFS-14403 Cost-Based RPC FairCallQueue Major ipc, namenode Erik Krogen Christopher Gregorian
HADOOP-16266 Add more fine-grained processing time metrics to the RPC layer Minor ipc Christopher Gregorian Erik Krogen
YARN-9629 Support configurable MIN_LOG_ROLLING_INTERVAL Minor log-aggregation, nodemanager, yarn Adam Antal Adam Antal
HDFS-13694 Making md5 computing being in parallel with image loading Major . zhouyingchao Lisheng Sun
HDFS-14632 Reduce useless #getNumLiveDataNodes call in SafeModeMonitor Major namenode He Xiaoqiao He Xiaoqiao
YARN-9573 DistributedShell cannot specify LogAggregationContext Major distributed-shell, log-aggregation, yarn Adam Antal Adam Antal
YARN-9337 GPU auto-discovery script runs even when the resource is given by hand Major yarn Adam Antal Adam Antal
YARN-9127 Create more tests to verify GpuDeviceInformationParser Major . Szilard Nemeth Peter Bacsko
YARN-9326 Fair Scheduler configuration defaults are not documented in case of min and maxResources Major docs, documentation, fairscheduler, yarn Adam Antal Adam Antal
HDFS-14547 DirectoryWithQuotaFeature.quota costs additional memory even the storage type quota is not set. Major . Jinglun Jinglun
HDFS-14693 NameNode should log a warning when EditLog IPC logger’s pending size exceeds limit. Minor namenode Xudong Cao Xudong Cao
YARN-9094 Remove unused interface method: NodeResourceUpdaterPlugin#handleUpdatedResourceFromRM Trivial . Szilard Nemeth Gergely Pollak
YARN-9096 Some GpuResourcePlugin and ResourcePluginManager methods are synchronized unnecessarily Major . Szilard Nemeth Gergely Pollak
YARN-9092 Create an object for cgroups mount enable and cgroups mount path as they belong together Minor . Szilard Nemeth Gergely Pollak
YARN-9124 Resolve contradiction in ResourceUtils: addMandatoryResources / checkMandatoryResources work differently Minor . Szilard Nemeth Adam Antal
YARN-8199 Logging fileSize of log files under NM Local Dir Major log-aggregation Prabhu Joseph Prabhu Joseph
YARN-9729 [UI2] Fix error message for logs when ATSv2 is offline Major yarn-ui-v2 Zoltan Siegl Zoltan Siegl
YARN-9135 NM State store ResourceMappings serialization are tested with Strings instead of real Device objects Major . Szilard Nemeth Peter Bacsko
HDFS-14370 Edit log tailing fast-path should allow for backoff Major namenode, qjm Erik Krogen Erik Krogen
YARN-9442 container working directory has group read permissions Minor yarn Jim Brennan Jim Brennan
HADOOP-16459 Backport [HADOOP-16266] “Add more fine-grained processing time metrics to the RPC layer” to branch-2 Major . Erik Krogen Erik Krogen
HDFS-14491 More Clarity on Namenode UI Around Blocks and Replicas Minor . Alan Jackoway Siyao Meng
YARN-9134 No test coverage for redefining FPGA / GPU resource types in TestResourceUtils Major . Szilard Nemeth Peter Bacsko
YARN-9133 Make tests more easy to comprehend in TestGpuResourceHandler Major . Szilard Nemeth Peter Bacsko
YARN-9140 Code cleanup in ResourcePluginManager.initialize and in TestResourcePluginManager Trivial . Szilard Nemeth Peter Bacsko
YARN-9676 Add DEBUG and TRACE level messages to AppLogAggregatorImpl and connected classes Major . Adam Antal Adam Antal
YARN-9488 Skip YARNFeatureNotEnabledException from ClientRMService Minor resourcemanager Prabhu Joseph Prabhu Joseph
YARN-8586 Extract log aggregation related fields and methods from RMAppImpl Major . Szilard Nemeth Peter Bacsko
YARN-9100 Add tests for GpuResourceAllocator and do minor code cleanup Major . Szilard Nemeth Peter Bacsko
HADOOP-15246 SpanReceiverInfo - Prefer ArrayList over LinkedList Trivial common David Mollitor David Mollitor
HADOOP-16158 DistCp to support checksum validation when copy blocks in parallel Major tools/distcp Kai Xie Kai Xie
HDFS-14746 Trivial test code update after HDFS-14687 Trivial ec Wei-Chiu Chuang kevin su
HDFS-13709 Report bad block to NN when transfer block encounter EIO exception Major datanode Chen Zhang Chen Zhang
HDFS-14665 HttpFS: LISTSTATUS response is missing HDFS-specific fields Major httpfs Siyao Meng Siyao Meng
HDFS-14276 [SBN read] Reduce tailing overhead Major ha, namenode Wei-Chiu Chuang Ayush Saxena
HADOOP-16061 Update Apache Yetus to 0.10.0 Major build Akira Ajisaka Akira Ajisaka
YARN-9756 Create metric that sums total memory/vcores preempted per round Major capacity scheduler Eric Payne Manikandan R
HDFS-14748 Make DataNodePeerMetrics#minOutlierDetectionSamples configurable Major . Lisheng Sun Lisheng Sun
HADOOP-15998 Ensure jar validation works on Windows. Blocker build Brian Grunkemeyer Brian Grunkemeyer
HDFS-14633 The StorageType quota and consume in QuotaFeature is not handled for rename Major . Jinglun Jinglun
YARN-9810 Add queue capacity/maxcapacity percentage metrics Major . Jonathan Hung Shubham Gupta
YARN-9763 Print application tags in application summary Major . Jonathan Hung Manoj Kumar
YARN-9795 ClusterMetrics to include AM allocation delay Minor . Fengnan Li Fengnan Li
YARN-8995 Log events info in AsyncDispatcher when event queue size cumulatively reaches a certain number every time. Major metrics, nodemanager, resourcemanager zhuqi zhuqi

BUG FIXES:

JIRA Summary Priority Component Reporter Contributor
HDFS-13973 getErasureCodingPolicy should log path in audit event Major hdfs Shweta Shweta
YARN-8868 Set HTTPOnly attribute to Cookie Major . Chandni Singh Chandni Singh
HDFS-14003 Fix findbugs warning in trunk for FSImageFormatPBINode Major . Yiqun Lin Yiqun Lin
YARN-8910 Misleading log statement in NM when max retries is -1 Minor . Chandni Singh Chandni Singh
YARN-7502 Nodemanager restart docs should describe nodemanager supervised property Major documentation Jason Lowe Suma Shivaprasad
YARN-8826 Fix lingering timeline collector after serviceStop in TimelineCollectorManager Trivial ATSv2 Prabha Manepalli Prabha Manepalli
HDFS-14021 TestReconstructStripedBlocksWithRackAwareness#testReconstructForNotEnoughRacks fails intermittently Major erasure-coding, test Xiao Chen Xiao Chen
MAPREDUCE-7151 RMContainerAllocator#handleJobPriorityChange expects application_priority always Major . Bibin A Chundatt Bilwa S T
HDFS-14028 HDFS OIV temporary dir deletes folder Major hdfs Adam Antal Adam Antal
HDFS-14027 DFSStripedOutputStream should implement both hsync methods Critical erasure-coding Xiao Chen Xiao Chen
HADOOP-15899 Update AWS Java SDK versions in NOTICE.txt Major . Akira Ajisaka Akira Ajisaka
HADOOP-15900 Update JSch versions in LICENSE.txt Major . Akira Ajisaka Akira Ajisaka
HDFS-14042 Fix NPE when PROVIDED storage is missing Major . Íñigo Goiri Virajith Jalaparti
HDFS-14043 Tolerate corrupted seen_txid file Major hdfs, namenode Lukas Majercak Lukas Majercak
YARN-8970 Improve the debug message in CS#allocateContainerOnSingleNode Trivial . Weiwei Yang Zhankun Tang
YARN-8865 RMStateStore contains large number of expired RMDelegationToken Major resourcemanager Wilfred Spiegelenburg Wilfred Spiegelenburg
HDFS-14048 DFSOutputStream close() throws exception on subsequent call after DataNode restart Major hdfs-client Erik Krogen Erik Krogen
MAPREDUCE-7156 NullPointerException when reaching max shuffle connections Major mrv2 Peter Bacsko Peter Bacsko
YARN-8866 Fix a parsing error for crossdomain.xml Major build, yarn-ui-v2 Takanobu Asanuma Takanobu Asanuma
HDFS-14039 ec -listPolicies doesn’t show correct state for the default policy when the default is not RS(6,3) Major erasure-coding Xiao Chen Kitti Nanasi
HADOOP-15916 Upgrade Maven Surefire plugin to 3.0.0-M1 Blocker build Akira Ajisaka Akira Ajisaka
YARN-9002 YARN Service keytab does not support s3, wasb, gs and is restricted to HDFS and local filesystem only Major yarn-native-services Gour Saha Gour Saha
YARN-8233 NPE in CapacityScheduler#tryCommit when handling allocate/reserve proposal whose allocatedOrReservedContainer is null Critical capacityscheduler Tao Yang Tao Yang
HDFS-14065 Failed Storage Locations shows nothing in the Datanode Volume Failures Major . Ayush Saxena Ayush Saxena
HADOOP-15923 create-release script should set max-cache-ttl as well as default-cache-ttl for gpg-agent Blocker build Akira Ajisaka Akira Ajisaka
HADOOP-15912 start-build-env.sh still creates an invalid /etc/sudoers.d/hadoop-build-${USER_ID} file entry after HADOOP-15802 Major build Akira Ajisaka Akira Ajisaka
HADOOP-15869 BlockDecompressorStream#decompress should not return -1 in case of IOException. Major . Surendra Singh Lilhore Surendra Singh Lilhore
MAPREDUCE-7158 Inefficient Flush Logic in JobHistory EventWriter Major . Zichen Sun Zichen Sun
HADOOP-15930 Exclude MD5 checksum files from release artifact Critical build Akira Ajisaka Akira Ajisaka
YARN-8856 TestTimelineReaderWebServicesHBaseStorage tests failing with NoClassDefFoundError Major . Jason Lowe Sushil Ks
HADOOP-15925 The config and log of gpg-agent are removed in create-release script Major build Akira Ajisaka Dinesh Chitlangia
HDFS-13963 NN UI is broken with IE11 Minor namenode, ui Daisuke Kobayashi Ayush Saxena
HDFS-14056 Fix error messages in HDFS-12716 Minor hdfs Adam Antal Ayush Saxena
YARN-8992 Fair scheduler can delete a dynamic queue while an application attempt is being added to the queue Major fairscheduler Haibo Chen Wilfred Spiegelenburg
YARN-8984 AMRMClient#OutstandingSchedRequests leaks when AllocationTags is null or empty Critical . Yang Wang Yang Wang
HADOOP-15948 Inconsistency in get and put syntax if filename/dirname contains space Minor fs vivek kumar Ayush Saxena
HDFS-13816 dfs.getQuotaUsage() throws NPE on non-existent dir instead of FileNotFoundException Major namenode Vinayakumar B Vinayakumar B
MAPREDUCE-7162 TestEvents#testEvents fails Critical jobhistoryserver, test Zhaohui Xin Zhaohui Xin
YARN-9056 Yarn Service Upgrade: Instance state changes from UPGRADING to READY without performing a readiness check Critical . Chandni Singh Chandni Singh
YARN-8812 Containers fail during creating a symlink which started with hyphen for a resource file Minor . Oleksandr Shevchenko Oleksandr Shevchenko
YARN-9030 Log aggregation changes to handle filesystems which do not support setting permissions Major log-aggregation Suma Shivaprasad Suma Shivaprasad
YARN-9067 YARN Resource Manager is running OOM because of leak of Configuration Object Major yarn-native-services Eric Yang Eric Yang
MAPREDUCE-7165 mapred-site.xml is misformatted in single node setup document Major documentation Akira Ajisaka Zhaohui Xin
HADOOP-15970 Upgrade plexus-utils from 2.0.5 to 3.1.0 Major security Akira Ajisaka Akira Ajisaka
HADOOP-15966 Hadoop Kerberos broken on macos as java.security.krb5.realm is reset: Null realm name (601) Major scripts Steve Loughran Steve Loughran
HADOOP-15974 Upgrade Curator version to 2.13.0 to fix ZK tests Major . Jason Lowe Akira Ajisaka
YARN-9071 NM and service AM don’t have updated status for reinitialized containers Critical . Billie Rinaldi Chandni Singh
MAPREDUCE-7159 FrameworkUploader: ensure proper permissions of generated framework tar.gz if restrictive umask is used Major mrv2 Peter Bacsko Peter Bacsko
YARN-9009 Fix flaky test TestEntityGroupFSTimelineStore.testCleanLogs Minor . OrDTesters OrDTesters
MAPREDUCE-7170 Doc typo in PluggableShuffleAndPluggableSort.md Minor documentation Zhaohui Xin Zhaohui Xin
YARN-9040 LevelDBCacheTimelineStore in ATS 1.5 leaks native memory Major timelineserver Tarun Parimi Tarun Parimi
YARN-9084 Service Upgrade: With default readiness check, the status of upgrade is reported to be successful prematurely Major . Chandni Singh Chandni Singh
HDFS-13661 Ls command with e option fails when the filesystem is not HDFS Major erasure-coding, tools Takanobu Asanuma Takanobu Asanuma
YARN-9154 Fix itemization in YARN service quickstart document Minor documentation Akira Ajisaka Ayush Saxena
HDFS-14166 Ls with -e option not giving the result in proper format Major . Soumyapn Shubham Dewan
HDFS-14046 In-Maintenance ICON is missing in datanode info page Major datanode Harshakiran Reddy Ranith Sardar
HDFS-14183 [SPS] Remove the -w parameter from the -satisfystoragepolicy usage Major . Ayush Saxena Ayush Saxena
YARN-9164 Shutdown NM may cause NPE when opportunistic container scheduling is enabled Critical . lujie lujie
YARN-8567 Fetching yarn logs fails for long running application if it is not present in timeline store Major log-aggregation Tarun Parimi Tarun Parimi
HADOOP-16028 Fix NetworkTopology chooseRandom function to support excluded nodes Major . Sihai Ke Sihai Ke
HADOOP-16030 AliyunOSS: bring fixes back from HADOOP-15671 Blocker fs/oss wujinhu wujinhu
YARN-9162 Fix TestRMAdminCLI#testHelp Major resourcemanager, test Ayush Saxena Ayush Saxena
YARN-9173 FairShare calculation broken for large values after YARN-8833 Major fairscheduler Wilfred Spiegelenburg Wilfred Spiegelenburg
YARN-8833 Avoid potential integer overflow when computing fair shares Major fairscheduler liyakun liyakun
HADOOP-16036 WASB: Disable jetty logging configuration announcement Major fs/azure Da Zhou Da Zhou
HADOOP-16016 TestSSLFactory#testServerWeakCiphers sporadically fails in precommit builds Major security, test Jason Lowe Akira Ajisaka
HDFS-14198 Upload and Create button doesn’t get enabled after getting reset. Major . Ayush Saxena Ayush Saxena
YARN-8747 [UI2] YARN UI2 page loading failed due to js error under some time zone configuration Critical webapp collinma collinma
YARN-9203 Fix typos in yarn-default.xml Trivial documentation Rahul Padmanabhan Rahul Padmanabhan
YARN-9194 Invalid event: REGISTERED and LAUNCH_FAILED at FAILED, and NullPointerException happens in RM while shutdown a NM Critical . lujie lujie
YARN-9204 RM fails to start if absolute resource is specified for partition capacity in CS queues Blocker yarn Jiandan Yang Jiandan Yang
HDFS-14207 ZKFC should catch exception when ha configuration missing Major hdfs Fei Hui Fei Hui
HDFS-14218 EC: Ls -e throw NPE when directory ec policy is disabled Major . Surendra Singh Lilhore Ayush Saxena
YARN-9210 RM nodes web page can not display node info Blocker yarn Jiandan Yang Jiandan Yang
YARN-9205 When using custom resource type, application will fail to run due to the CapacityScheduler throws InvalidResourceRequestException(GREATER_THEN_MAX_ALLOCATION) Critical . Zhankun Tang Zhankun Tang
YARN-8961 [UI2] Flow Run End Time shows ‘Invalid date’ Major . Charan Hebri Akhil PB
HADOOP-16065 -Ddynamodb should be -Ddynamo in AWS SDK testing document Minor documentation Akira Ajisaka Akira Ajisaka
HDFS-14228 Incorrect getSnapshottableDirListing() javadoc Major snapshots Wei-Chiu Chuang Dinesh Chitlangia
YARN-9222 Print launchTime in ApplicationSummary Major . Jonathan Hung Jonathan Hung
YARN-8901 Restart “NEVER” policy does not work with component dependency Critical . Yesha Vora Suma Shivaprasad
YARN-9237 NM should ignore sending finished apps to RM during RM fail-over Major yarn Jiandan Yang Jiandan Yang
YARN-6616 YARN AHS shows submitTime for jobs same as startTime Minor . Prabhu Joseph Prabhu Joseph
YARN-9099 GpuResourceAllocator#getReleasingGpus calculates number of GPUs in a wrong way Major . Szilard Nemeth Szilard Nemeth
YARN-9262 TestRMAppAttemptTransitions is failing with an NPE Critical resourcemanager Sunil Govindan lujie
HDFS-14232 libhdfs is not included in binary tarball Critical build, libhdfs Akira Ajisaka Akira Ajisaka
MAPREDUCE-7177 Disable speculative execution in TestDFSIO Major . Kihwal Lee Zhaohui Xin
HADOOP-16076 SPNEGO+SSL Client Connections with HttpClient Broken Major build, security Larry McCay Larry McCay
HADOOP-16074 WASB: Update container not found error code Major fs/azure Da Zhou Da Zhou
YARN-8498 Yarn NodeManager OOM Listener Fails Compilation on Ubuntu 18.04 Blocker . Jack Bearden Ayush Saxena
YARN-9206 RMServerUtils does not count SHUTDOWN as an accepted state Major . Kuhu Shukla Kuhu Shukla
HADOOP-16032 Distcp It should clear sub directory ACL before applying new ACL on it. Major tools/distcp Ranith Sardar Ranith Sardar
HDFS-14140 JournalNodeSyncer authentication is failing in secure cluster Major journal-node, security Surendra Singh Lilhore Surendra Singh Lilhore
YARN-9257 Distributed Shell client throws a NPE for a non-existent queue Major distributed-shell Charan Hebri Charan Hebri
YARN-8761 Service AM support for decommissioning component instances Major . Billie Rinaldi Billie Rinaldi
HDFS-14266 EC : Fsck -blockId shows null for EC Blocks if One Block Is Not Available. Major . Harshakiran Reddy Ayush Saxena
HDFS-14274 EC: NPE While Listing EC Policy For A Directory Following Replication Policy. Major erasure-coding Souryakanta Dwivedy Ayush Saxena
HDFS-14263 Remove unnecessary block file exists check from FsDatasetImpl#getBlockInputStream() Major datanode Surendra Singh Lilhore Surendra Singh Lilhore
YARN-7761 [UI2] Clicking ‘master container log’ or ‘Link’ next to ‘log’ under application’s appAttempt goes to Old UI’s Log link Major yarn-ui-v2 Sumana Sathish Akhil PB
YARN-9295 [UI2] Fix label typo in Cluster Overview page Trivial yarn-ui-v2 Charan Hebri Charan Hebri
YARN-9308 fairscheduler-statedump.log gets generated regardless of service again after the merge of HDFS-7240 Blocker fairscheduler, scheduler Akira Ajisaka Wilfred Spiegelenburg
YARN-9284 Fix the unit of yarn.service.am-resource.memory in the document Minor documentation, yarn-native-services Masahiro Tanaka Masahiro Tanaka
YARN-9283 Javadoc of LinuxContainerExecutor#addSchedPriorityCommand has a wrong property name as reference Minor documentation Szilard Nemeth Adam Antal
YARN-9286 [Timeline Server] Sorting based on FinalStatus shows pop-up message Minor timelineserver Nallasivan Bilwa S T
HDFS-14081 hdfs dfsadmin -metasave metasave_test results NPE Major hdfs Shweta Shweta
HADOOP-15813 Enable more reliable SSL connection reuse Major common Daryn Sharp Daryn Sharp
HDFS-14216 NullPointerException happens in NamenodeWebHdfs Critical . lujie lujie
HADOOP-16105 WASB in secure mode does not set connectingUsingSAS Major fs/azure Steve Loughran Steve Loughran
YARN-9238 Avoid allocating opportunistic containers to previous/removed/non-exist application attempt Critical . lujie lujie
YARN-9118 Handle exceptions with parsing user defined GPU devices in GpuDiscoverer Major . Szilard Nemeth Szilard Nemeth
YARN-9317 Avoid repeated YarnConfiguration#timelineServiceV2Enabled check Major . Bibin A Chundatt Prabhu Joseph
YARN-9213 RM Web UI v1 does not show custom resource allocations for containers page Major . Szilard Nemeth Szilard Nemeth
YARN-9329 updatePriority is blocked when using FairScheduler Major . Jiandan Yang Jiandan Yang
YARN-9248 RMContainerImpl:Invalid event: ACQUIRED at KILLED Major . lujie lujie
HADOOP-16018 DistCp won’t reassemble chunks when blocks per chunk > 0 Major tools/distcp Kai Xie Kai Xie
YARN-9334 YARN Service Client does not work with SPNEGO when knox is configured Major yarn-native-services Tarun Parimi Billie Rinaldi
HDFS-14305 Serial number in BlockTokenSecretManager could overlap between different namenodes Major namenode, security Chao Sun He Xiaoqiao
HDFS-14314 fullBlockReportLeaseId should be reset after registering to NN Critical datanode star star
YARN-8803 [UI2] Show flow runs in the order of recently created time in graph widgets Major yarn-ui-v2 Akhil PB Akhil PB
HDFS-14317 Standby does not trigger edit log rolling when in-progress edit log tailing is enabled Critical . Ekanth Sethuramalingam Ekanth Sethuramalingam
HDFS-14333 Datanode fails to start if any disk has errors during Namenode registration Major datanode Stephen O’Donnell Stephen O’Donnell
HADOOP-16192 CallQueue backoff bug fixes: doesn’t perform backoff when add() is used, and doesn’t update backoff when refreshed Major ipc Erik Krogen Erik Krogen
HDFS-14037 Fix SSLFactory truststore reloader thread leak in URLConnectionFactory Major hdfs-client, webhdfs Takanobu Asanuma Takanobu Asanuma
HADOOP-16225 Fix links to the developer mailing lists in DownstreamDev.md Minor documentation Akira Ajisaka Wanqiang Ji
HADOOP-16226 new Path(String str) does not remove all the trailing slashes of str Minor fs Akira Ajisaka Akira Ajisaka
HADOOP-16232 Fix errors in the checkstyle configration xmls Major build Akira Ajisaka Wanqiang Ji
HDFS-14389 getAclStatus returns incorrect permissions and owner when an iNodeAttributeProvider is configured Major namenode Stephen O’Donnell Stephen O’Donnell
HDFS-14407 Fix misuse of SLF4j logging API in DatasetVolumeChecker#checkAllVolumes Minor . Wanqiang Ji Wanqiang Ji
YARN-9413 Queue resource leak after app fail for CapacityScheduler Major capacityscheduler Tao Yang Tao Yang
HADOOP-14635 Javadoc correction for AccessControlList#buildACL Minor documentation Bibin A Chundatt Yeliang Cang
HADOOP-14544 DistCp documentation for command line options is misaligned. Minor documentation Chris Nauroth Masatake Iwasaki
HDFS-10477 Stop decommission a rack of DataNodes caused NameNode fail over to standby Major namenode yunjiong zhao yunjiong zhao
YARN-9487 NodeManager native build shouldn’t link against librt on macOS Major nodemanager Siyao Meng Siyao Meng
YARN-6695 Race condition in RM for publishing container events vs appFinished events causes NPE Critical . Rohith Sharma K S Prabhu Joseph
YARN-8622 NodeManager native build fails due to getgrouplist not found on macOS Major nodemanager Ewan Higgs Siyao Meng
HADOOP-16265 Configuration#getTimeDuration is not consistent between default value and manual settings. Major . star star
HDFS-13677 Dynamic refresh Disk configuration results in overwriting VolumeMap Blocker . xuzq xuzq
YARN-9285 RM UI progress column is of wrong type Minor yarn Ahmed Hussein Ahmed Hussein
HADOOP-16278 With S3A Filesystem, Long Running services End up Doing lot of GC and eventually die Major common, hadoop-aws, metrics Rajat Khandelwal Rajat Khandelwal
YARN-9504 [UI2] Fair scheduler queue view page does not show actual capacity Major fairscheduler, yarn-ui-v2 Zoltan Siegl Zoltan Siegl
YARN-9519 TFile log aggregation file format is not working for yarn.log-aggregation.TFile.remote-app-log-dir config Major log-aggregation Adam Antal Adam Antal
YARN-9508 YarnConfiguration areNodeLabel enabled is costly in allocation flow Critical . Bibin A Chundatt Bilwa S T
HADOOP-16247 NPE in FsUrlConnection Major hdfs-client Karthik Palanisamy Karthik Palanisamy
HADOOP-16248 MutableQuantiles leak memory under heavy load Major metrics Alexis Daboville Alexis Daboville
HDFS-14323 Distcp fails in Hadoop 3.x when 2.x source webhdfs url has special characters in hdfs file path Major webhdfs Srinivasu Majeti Srinivasu Majeti
MAPREDUCE-7205 Treat container scheduler kill exit code as a task attempt killing event Major applicationmaster, mr-am, mrv2 Wanqiang Ji Wanqiang Ji
HDFS-14500 NameNode StartupProgress continues to report edit log segments after the LOADING_EDITS phase is finished Major namenode Erik Krogen Erik Krogen
YARN-9500 Fix typos in ResourceModel.md Trivial documentation leiqiang leiqiang
HADOOP-16331 Fix ASF License check in pom.xml Major . Wanqiang Ji Akira Ajisaka
YARN-9542 Fix LogsCLI guessAppOwner ignores custom file format suffix Minor log-aggregation Prabhu Joseph Prabhu Joseph
HDFS-14512 ONE_SSD policy will be violated while write data with DistributedFileSystem.create(….favoredNodes) Major . Shen Yinjie Ayush Saxena
HADOOP-16334 Fix yetus-wrapper not working when HADOOP_YETUS_VERSION >= 0.9.0 Major yetus Wanqiang Ji Wanqiang Ji
HDFS-14521 Suppress setReplication logging. Major . Kihwal Lee Kihwal Lee
YARN-9507 Fix NPE in NodeManager#serviceStop on startup failure Minor . Bilwa S T Bilwa S T
YARN-8947 [UI2] Active User info missing from UI2 Major yarn-ui-v2 Akhil PB Akhil PB
YARN-8906 [UI2] NM hostnames not displayed correctly in Node Heatmap Chart Major . Charan Hebri Akhil PB
YARN-9580 Fulfilled reservation information in assignment is lost when transferring in ParentQueue#assignContainers Major capacityscheduler Tao Yang Tao Yang
YARN-8625 Aggregate Resource Allocation for each job is not present in ATS Major ATSv2 Prabhu Joseph Prabhu Joseph
HADOOP-16345 Potential NPE when instantiating FairCallQueue metrics Major ipc Erik Krogen Erik Krogen
YARN-9594 Fix missing break statement in ContainerScheduler#handle Major . lujie lujie
YARN-9565 RMAppImpl#ranNodes not cleared on FinalTransition Major . Bibin A Chundatt Bilwa S T
YARN-9547 ContainerStatusPBImpl default execution type is not returned Major . Bibin A Chundatt Bilwa S T
HDFS-13231 Extend visualization for Decommissioning, Maintenance Mode under Datanode tab in the NameNode UI Major datanode, namenode Haibo Yan Stephen O’Donnell
HDFS-14535 The default 8KB buffer in requestFileDescriptors#BufferedOutputStream is causing lots of heap allocation in HBase when using short-circut read Major hdfs-client Zheng Hu Zheng Hu
HDFS-13730 BlockReaderRemote.sendReadResult throws NPE Major hdfs-client Wei-Chiu Chuang Yuanbo Liu
YARN-9584 Should put initializeProcessTrees method call before get pid Critical nodemanager Wanqiang Ji Wanqiang Ji
HDFS-14010 Pass correct DF usage to ReservedSpaceCalculator builder Minor . Lukas Majercak Lukas Majercak
HDFS-14078 Admin helper fails to prettify NullPointerExceptions Major . Elek, Marton Elek, Marton
HDFS-14101 Random failure of testListCorruptFilesCorruptedBlock Major test Kihwal Lee Zsolt Venczel
HDFS-14465 When the Block expected replications is larger than the number of DataNodes, entering maintenance will never exit. Major . Yicong Cai Yicong Cai
HDFS-13893 DiskBalancer: no validations for Disk balancer commands Major diskbalancer Harshakiran Reddy Lokesh Jain
YARN-9209 When nodePartition is not set in Placement Constraints, containers are allocated only in default partition Major capacity scheduler, scheduler Tarun Parimi Tarun Parimi
HDFS-12487 FsDatasetSpi.isValidBlock() lacks null pointer check inside and neither do the callers Major balancer & mover, diskbalancer liumi liumi
HDFS-14074 DataNode runs async disk checks maybe throws NullPointerException, and DataNode failed to register to NameSpace. Major hdfs guangyi lu guangyi lu
HDFS-14541 When evictableMmapped or evictable size is zero, do not throw NoSuchElementException Major hdfs-client, performance Zheng Hu Lisheng Sun
HDFS-14598 Findbugs warning caused by HDFS-12487 Minor diskbalancer Wei-Chiu Chuang He Xiaoqiao
YARN-9639 DecommissioningNodesWatcher cause memory leak Blocker . Bibin A Chundatt Bilwa S T
YARN-9581 Fix WebAppUtils#getRMWebAppURLWithScheme ignores rm2 Major client Prabhu Joseph Prabhu Joseph
YARN-9327 Improve synchronisation in ProtoUtils#convertToProtoFormat block Critical . Bibin A Chundatt Bibin A Chundatt
YARN-9655 AllocateResponse in FederationInterceptor lost applicationPriority Major federation hunshenshi hunshenshi
HADOOP-16385 Namenode crashes with “RedundancyMonitor thread received Runtime exception” Major . krishna reddy Ayush Saxena
YARN-9644 First RMContext object is always leaked during switch over Blocker . Bibin A Chundatt Bibin A Chundatt
HDFS-14629 Property value Hard Coded in DNConf.java Trivial . hemanthboyina hemanthboyina
YARN-9557 Application fails in diskchecker when ReadWriteDiskValidator is configured. Critical nodemanager Anuruddh Nayak Bilwa S T
HDFS-12703 Exceptions are fatal to decommissioning monitor Critical namenode Daryn Sharp He Xiaoqiao
HDFS-12748 NameNode memory leak when accessing webhdfs GETHOMEDIRECTORY Major hdfs Jiandan Yang Weiwei Yang
YARN-9625 UI2 - No link to a queue on the Queues page for Fair Scheduler Major . Charan Hebri Zoltan Siegl
HDFS-14466 Add a regression test for HDFS-14323 Minor fs, test, webhdfs Yuya Ebihara Masatake Iwasaki
YARN-9235 If linux container executor is not set for a GPU cluster GpuResourceHandlerImpl is not initialized and NPE is thrown Major yarn Antal Bálint Steinbach Adam Antal
YARN-9626 UI2 - Fair scheduler queue apps page issues Major . Charan Hebri Zoltan Siegl
YARN-9645 Fix Invalid event FINISHED_CONTAINERS_PULLED_BY_AM at NEW on NM restart Major . krishna reddy Bilwa S T
YARN-9682 Wrong log message when finalizing the upgrade Trivial . kyungwan nam kyungwan nam
HADOOP-16440 Distcp can not preserve timestamp with -delete option Major . ludun ludun
MAPREDUCE-7076 TestNNBench#testNNBenchCreateReadAndDelete failing in our internal build Minor test Rushabh S Shah kevin su
HADOOP-16443 Improve help text for setfacl –set option Minor fs Stephen O’Donnell Stephen O’Donnell
YARN-9668 UGI conf doesn’t read user overridden configurations on RM and NM startup Major . Jonathan Hung Jonathan Hung
HADOOP-9844 NPE when trying to create an error message response of SASL RPC Major ipc Steve Loughran Steve Loughran
HADOOP-16245 Enabling SSL within LdapGroupsMapping can break system SSL configs Major common, security Erik Krogen Erik Krogen
HDFS-14429 Block remain in COMMITTED but not COMPLETE caused by Decommission Major . Yicong Cai Yicong Cai
HADOOP-16435 RpcMetrics should not be retained forever Critical rpc-server Zoltan Haindrich Zoltan Haindrich
YARN-9596 QueueMetrics has incorrect metrics when labelled partitions are involved Major capacity scheduler Muhammad Samir Khan Muhammad Samir Khan
MAPREDUCE-7225 Fix broken current folder expansion during MR job start Major mrv2 Adam Antal Peter Bacsko
HDFS-13529 Fix default trash policy emptier trigger time correctly Major namenode He Xiaoqiao He Xiaoqiao
HADOOP-15681 AuthenticationFilter should generate valid date format for Set-Cookie header regardless of default Locale Minor security Cao Manh Dat Cao Manh Dat
HDFS-14685 DefaultAuditLogger doesn’t print CallerContext Major hdfs xuzq xuzq
HDFS-14462 WebHDFS throws “Error writing request body to server” instead of DSQuotaExceededException Major webhdfs Erik Krogen Simbarashe Dzinamarira
HDFS-14691 Wrong usage hint for hadoop fs command test Minor hdfs Jianfei Jiang Jianfei Jiang
HDFS-14557 JournalNode error: Can’t scan a pre-transactional edit log Major ha Wei-Chiu Chuang Stephen O’Donnell
HDFS-14692 Upload button should not encode complete url Major . Lokesh Jain Lokesh Jain
HADOOP-15908 hadoop-build-tools jar is downloaded from remote repository instead of using from local Minor build Oleksandr Shevchenko Oleksandr Shevchenko
HDFS-14631 The DirectoryScanner doesn’t fix the wrongly placed replica. Major . Jinglun Jinglun
YARN-9685 NPE when rendering the info table of leaf queue in non-accessible partitions Major capacityscheduler Tao Yang Tao Yang
HDFS-14459 ClosedChannelException silently ignored in FsVolumeList.addBlockPool() Major datanode Stephen O’Donnell Stephen O’Donnell
HDFS-13359 DataXceiver hung due to the lock in FsDatasetImpl#getBlockInputStream Major datanode Yiqun Lin Yiqun Lin
YARN-9451 AggregatedLogsBlock shows wrong NM http port Minor nodemanager Prabhu Joseph Prabhu Joseph
YARN-9723 ApplicationPlacementContext is not required for terminated jobs during recovery Major resourcemanager Prabhu Joseph Prabhu Joseph
HDFS-12914 Block report leases cause missing blocks until next report Critical namenode Daryn Sharp Santosh Marella
HDFS-14148 HDFS OIV ReverseXML SnapshotSection parser throws exception when there are more than one snapshottable directory Major hdfs Siyao Meng Siyao Meng
HDFS-14595 HDFS-11848 breaks API compatibility Blocker . Wei-Chiu Chuang Siyao Meng
HDFS-14423 Percent (%) and plus (+) characters no longer work in WebHDFS Major webhdfs Jing Wang Masatake Iwasaki
MAPREDUCE-7230 TestHSWebApp.testLogsViewSingle fails Major jobhistoryserver, test Prabhu Joseph Prabhu Joseph
YARN-9749 TestAppLogAggregatorImpl#testDFSQuotaExceeded fails on trunk Major log-aggregation, test Peter Bacsko Adam Antal
HDFS-14687 Standby Namenode never come out of safemode when EC files are being written. Critical ec, namenode Surendra Singh Lilhore Surendra Singh Lilhore
HDFS-13101 Yet another fsimage corruption related to snapshot Major snapshots Yongjun Zhang Shashikant Banerjee
HDFS-13201 Fix prompt message in testPolicyAndStateCantBeNull Minor . chencan chencan
HDFS-14311 Multi-threading conflict at layoutVersion when loading block pool storage Major rolling upgrades Yicong Cai Yicong Cai
HDFS-14582 Failed to start DN with ArithmeticException when NULL checksum used Major datanode Surendra Singh Lilhore Surendra Singh Lilhore
YARN-9217 Nodemanager will fail to start if GPU is misconfigured on the node or GPU drivers missing Major yarn Antal Bálint Steinbach Peter Bacsko
HADOOP-16494 Add SHA-256 or SHA-512 checksum to release artifacts to comply with the release distribution policy Blocker build Akira Ajisaka Akira Ajisaka
YARN-9774 Fix order of arguments for assertEquals in TestSLSUtils Minor test Nikhil Navadiya Nikhil Navadiya
HDFS-13596 NN restart fails after RollingUpgrade from 2.x to 3.x Blocker hdfs Hanisha Koneru Fei Hui
HDFS-14396 Failed to load image from FSImageFile when downgrade from 3.x to 2.x Blocker rolling upgrades Fei Hui Fei Hui
YARN-8917 Absolute (maximum) capacity of level3+ queues is wrongly calculated for absolute resource Critical capacityscheduler Tao Yang Tao Yang
YARN-9642 Fix Memory Leak in AbstractYarnScheduler caused by timer Blocker resourcemanager Bibin A Chundatt Bibin A Chundatt
HDFS-13977 NameNode can kill itself if it tries to send too many txns to a QJM simultaneously Major namenode, qjm Erik Krogen Erik Krogen
HDFS-2470 NN should automatically set permissions on dfs.namenode.*.dir Major namenode Aaron T. Myers Siddharth Wagle
YARN-9438 launchTime not written to state store for running applications Major . Jonathan Hung Jonathan Hung
YARN-9640 Slow event processing could cause too many attempt unregister events Critical . Bibin A Chundatt Bibin A Chundatt
HDFS-12212 Options.Rename.To_TRASH is considered even when Options.Rename.NONE is specified Major namenode Vinayakumar B Vinayakumar B
YARN-9714 ZooKeeper connection in ZKRMStateStore leaks after RM transitioned to standby Major resourcemanager Tao Yang Tao Yang
HDFS-8178 QJM doesn’t move aside stale inprogress edits files Major qjm Zhe Zhang Istvan Fajth
HDFS-14706 Checksums are not checked if block meta file is less than 7 bytes Major . Stephen O’Donnell Stephen O’Donnell
YARN-9797 LeafQueue#activateApplications should use resourceCalculator#fitsIn Blocker . Bibin A Chundatt Bilwa S T
YARN-9785 Fix DominantResourceCalculator when one resource is zero Blocker . Bilwa S T Bilwa S T
YARN-9718 Yarn REST API, services endpoint remote command ejection Major . Eric Yang Eric Yang
HADOOP-16255 ChecksumFS.Make FileSystem.rename(path, path, options) doesn’t rename checksum Major fs Steve Loughran Jungtaek Lim
YARN-9817 Fix failing testcases due to not initialized AsyncDispatcher - ArithmeticException: / by zero Major test Prabhu Joseph Prabhu Joseph
YARN-9813 RM does not start on JDK11 when UIv2 is enabled Critical resourcemanager, yarn Adam Antal Adam Antal
YARN-9820 RM logs InvalidStateTransitionException when app is submitted Critical . Rohith Sharma K S Prabhu Joseph

TESTS:

JIRA Summary Priority Component Reporter Contributor
YARN-8907 Modify a logging message in TestCapacityScheduler Trivial . Zhankun Tang Zhankun Tang
YARN-8904 TestRMDelegationTokens can fail in testRMDTMasterKeyStateOnRollingMasterKey Minor test Wilfred Spiegelenburg Wilfred Spiegelenburg
YARN-8944 TestContainerAllocation.testUserLimitAllocationMultipleContainers failure after YARN-8896 Minor capacity scheduler Wilfred Spiegelenburg Wilfred Spiegelenburg
YARN-9263 TestConfigurationNodeAttributesProvider fails after Mockito updated Minor . Weiwei Yang Weiwei Yang
YARN-9315 TestCapacitySchedulerMetrics fails intermittently Minor capacity scheduler Prabhu Joseph Prabhu Joseph
YARN-9316 TestPlacementConstraintsUtil#testInterAppConstraintsByAppID fails intermittently Minor capacity scheduler Prabhu Joseph Prabhu Joseph
YARN-9324 TestSchedulingRequestContainerAllocation(Async) fails with junit-4.11 Major capacity scheduler Prabhu Joseph Prabhu Joseph
YARN-9325 TestQueueManagementDynamicEditPolicy fails intermittent Minor capacity scheduler Prabhu Joseph Prabhu Joseph
HDFS-11950 Disable libhdfs zerocopy test on Mac Minor libhdfs John Zhuge Akira Ajisaka

SUB-TASKS:

JIRA Summary Priority Component Reporter Contributor
YARN-6989 Ensure timeline service v2 codebase gets UGI from HttpServletRequest in a consistent way Major timelineserver Vrushali C Abhishek Modi
YARN-8834 Provide Java client for fetching Yarn specific entities from TimelineReader Critical timelinereader Rohith Sharma K S Abhishek Modi
YARN-3879 [Storage implementation] Create HDFS backing storage implementation for ATS reads Major timelineserver Tsuyoshi Ozawa Abhishek Modi
YARN-6098 Add documentation for Delete Queue Major capacity scheduler, documentation Naganarasimha G R Suma Shivaprasad
YARN-8456 Fix a configuration handling bug when user leave FPGA discover executable path configuration default but set OpenCL SDK path environment variable Major yarn Zhankun Tang Zhankun Tang
HADOOP-15868 AliyunOSS: update document for properties of multiple part download, multiple part upload and directory copy Major fs/oss wujinhu wujinhu
YARN-8871 Document behavior of YARN-5742 Major . Vrushali C Suma Shivaprasad
YARN-7754 [Atsv2] Update document for running v1 and v2 TS Major . Rohith Sharma K S Suma Shivaprasad
HDFS-14047 [libhdfs++] Fix hdfsGetLastExceptionRootCause bug in test_libhdfs_threaded.c Major libhdfs, native Anatoli Shein Anatoli Shein
YARN-8988 Reduce the verbose log on RM heartbeat path when distributed node-attributes is enabled Major . Weiwei Yang Tao Yang
HADOOP-15846 ABFS: fix mask related bugs in setAcl, modifyAclEntries and removeAclEntries. Major fs/azure Thomas Marquardt junhua gu
HADOOP-15812 ABFS: Improve AbfsRestOperationException format to ensure full msg can be displayed on console Major fs/azure Da Zhou Da Zhou
YARN-8987 Usability improvements node-attributes CLI Critical . Weiwei Yang Bibin A Chundatt
HADOOP-15876 Use keySet().removeAll() to remove multiple keys from Map in AzureBlobFileSystemStore Minor fs/azure Ted Yu Da Zhou
HADOOP-15917 AliyunOSS: fix incorrect ReadOps and WriteOps in statistics Major fs/oss wujinhu wujinhu
YARN-8303 YarnClient should contact TimelineReader for application/attempt/container report Critical . Rohith Sharma K S Abhishek Modi
HADOOP-15872 ABFS: Update to target 2018-11-09 REST version for ADLS Gen 2 Major fs/azure Thomas Marquardt junhua gu
HADOOP-15940 ABFS: For HNS account, avoid unnecessary get call when doing Rename Major fs/azure Da Zhou Da Zhou
YARN-8986 publish all exposed ports to random ports when using bridge network Minor yarn dockerzhang dockerzhang
HADOOP-15932 Oozie unable to create sharelib in s3a filesystem Critical fs, fs/s3 Soumitra Sulav Steve Loughran
YARN-9034 ApplicationCLI should have option to take clusterId Major . Rohith Sharma K S Rohith Sharma K S
HDFS-13713 Add specification of Multipart Upload API to FS specification, with contract tests Blocker fs, test Steve Loughran Ewan Higgs
HADOOP-15968 ABFS: add try catch for UGI failure when initializing ABFS Major fs/azure Da Zhou Da Zhou
HADOOP-15969 ABFS: getNamespaceEnabled can fail blocking user access thru ACLs Major fs/azure Da Zhou Da Zhou
HADOOP-15972 ABFS: reduce list page size to to 500 Major fs/azure Da Zhou Da Zhou
HADOOP-16004 ABFS: Convert 404 error response in AbfsInputStream and AbfsOutPutStream to FileNotFoundException Major fs/azure Da Zhou Da Zhou
YARN-9126 Container reinit always fails in branch-3.2 and trunk Major . Eric Yang Chandni Singh
YARN-8925 Updating distributed node attributes only when necessary Major resourcemanager Tao Yang Tao Yang
HADOOP-16009 Replace the url of the repository in Apache Hadoop source code Major documentation Akira Ajisaka Akira Ajisaka
HADOOP-15323 AliyunOSS: Improve copy file performance for AliyunOSSFileSystemStore Major fs/oss wujinhu wujinhu
YARN-6149 Allow port range to be specified while starting NM Timeline collector manager. Major timelineserver Varun Saxena Abhishek Modi
HADOOP-16040 ABFS: Bug fix for tolerateOobAppends configuration Major fs/azure Da Zhou Da Zhou
HADOOP-15975 ABFS: remove timeout check for DELETE and RENAME Major fs/azure Da Zhou Da Zhou
HADOOP-15662 ABFS: Better exception handling of DNS errors Major fs/azure Thomas Marquardt Da Zhou
HADOOP-16045 Don’t run TestDU on Windows Trivial common, test Lukas Majercak Lukas Majercak
HADOOP-16044 ABFS: Better exception handling of DNS errors followup Major . Da Zhou Da Zhou
HADOOP-16048 ABFS: Fix Date format parser Major fs/azure Da Zhou Da Zhou
YARN-8101 Add UT to verify node-attributes in RM nodes rest API Minor resourcemanager, restapi Weiwei Yang Prabhu Joseph
HADOOP-16041 UserAgent string for ABFS Major fs/azure Shweta Shweta
HADOOP-16079 Token.toString faulting if any token listed can’t load. Blocker security Steve Loughran Steve Loughran
YARN-9275 Add link to NodeAttributes doc in PlacementConstraints document Minor documentation Weiwei Yang Masatake Iwasaki
YARN-6735 Have a way to turn off container metrics from NMs Major timelineserver Vrushali C Abhishek Modi
HADOOP-15954 ABFS: Enable owner and group conversion for MSI and login user using OAuth Major fs/azure junhua gu Da Zhou
YARN-9253 Add UT to verify Placement Constraint in Distributed Shell Major . Prabhu Joseph Prabhu Joseph
YARN-9252 Allocation Tag Namespace support in Distributed Shell Major distributed-shell Prabhu Joseph Prabhu Joseph
YARN-8555 Parameterize TestSchedulingRequestContainerAllocation(Async) to cover both PC handler options Minor . Weiwei Yang Prabhu Joseph
YARN-9293 Optimize MockAMLauncher event handling Major . Bibin A Chundatt Bibin A Chundatt
HADOOP-16104 Wasb tests to downgrade to skip when test a/c is namespace enabled Major fs/azure, test Steve Loughran Masatake Iwasaki
YARN-9258 Support to specify allocation tags without constraint in distributed shell CLI Major distributed-shell Prabhu Joseph Prabhu Joseph
HADOOP-16136 ABFS: Should only transform username to short name Major . Da Zhou Da Zhou
YARN-5336 Limit the flow name size & consider cleanup for hex chars Major timelineserver Vrushali C Sushil Ks
YARN-3841 [Storage implementation] Adding retry semantics to HDFS backing storage Major timelineserver Tsuyoshi Ozawa Abhishek Modi
HADOOP-16169 ABFS: Bug fix for getPathProperties Major fs/azure Da Zhou Da Zhou
HADOOP-16109 Parquet reading S3AFileSystem causes EOF Blocker fs/s3 Dave Christianson Steve Loughran
HADOOP-15625 S3A input stream to use etags/version number to detect changed source files Major fs/s3 Brahma Reddy Battula Ben Roling
HADOOP-16124 Extend documentation in testing.md about endpoint constants Trivial hadoop-aws Adam Antal Adam Antal
HADOOP-16191 AliyunOSS: improvements for copyFile/copyDirectory and logging Major fs/oss wujinhu wujinhu
YARN-9387 Update document for ATS HBase Custom tablenames (-entityTableName) Critical ATSv2 Prabhu Joseph Prabhu Joseph
YARN-9389 FlowActivity and FlowRun table prefix is wrong Minor ATSv2 Prabhu Joseph Prabhu Joseph
YARN-9391 Disable PATH variable to be passed to Docker container Major . Eric Yang Jim Brennan
HADOOP-16058 S3A tests to include Terasort Major fs/s3, test Steve Loughran Steve Loughran
HADOOP-16220 Add findbugs ignores for unjustified issues during update to guava to 27.0-jre in hadoop-project Major . Gabor Bota Gabor Bota
YARN-9418 ATSV2 /apps/appId/entities/YARN_CONTAINER rest api does not show metrics Critical ATSv2 Prabhu Joseph Prabhu Joseph
HADOOP-16233 S3AFileStatus to declare that isEncrypted() is always true Minor fs/s3 Steve Loughran Steve Loughran
YARN-9303 Username splits won’t help timelineservice.app_flow table Major ATSv2 Prabhu Joseph Prabhu Joseph
YARN-9382 Publish container killed, paused and resumed events to ATSv2. Major . Abhishek Modi Abhishek Modi
YARN-9335 [atsv2] Restrict the number of elements held in timeline collector when backend is unreachable for async calls Major . Vrushali C Abhishek Modi
HADOOP-16269 ABFS: add listFileStatus with StartFrom Major fs/azure Da Zhou Da Zhou
HADOOP-16251 ABFS: add FSMainOperationsBaseTest Major fs/azure Da Zhou Da Zhou
HADOOP-16306 AliyunOSS: Remove temporary files when upload small files to OSS Major fs/oss wujinhu wujinhu
YARN-7537 [Atsv2] load hbase configuration from filesystem rather than URL Major . Rohith Sharma K S Prabhu Joseph
HDFS-14553 Make queue size of BlockReportProcessingThread configurable Major namenode He Xiaoqiao He Xiaoqiao
HADOOP-16211 Update guava to 27.0-jre in hadoop-project branch-3.2 Major . Gabor Bota Gabor Bota
YARN-8499 ATS v2 Generic TimelineStorageMonitor Major ATSv2 Sunil Govindan Prabhu Joseph
YARN-9374 HBaseTimelineWriterImpl sync writes has to avoid thread blocking if storage down Major ATSv2 Prabhu Joseph Prabhu Joseph
HADOOP-16401 ABFS: port Azure doc to 3.2 branch Major fs/azure Da Zhou Masatake Iwasaki
HDFS-14034 Support getQuotaUsage API in WebHDFS Major fs, webhdfs Erik Krogen Chao Sun
YARN-9765 SLS runner crashes when run with metrics turned off. Major . Abhishek Modi Abhishek Modi
HDFS-14674 [SBN read] Got an unexpected txid when tail editlog Blocker . wangzhaohui wangzhaohui
YARN-9775 RMWebServices /scheduler-conf GET returns all hadoop configurations for ZKConfigurationStore Major restapi Prabhu Joseph Prabhu Joseph
HDFS-14779 Fix logging error in TestEditLog#testMultiStreamsLoadEditWithConfMaxTxns Major . Jonathan Hung Jonathan Hung
YARN-9804 Update ATSv2 document for latest feature supports Blocker . Rohith Sharma K S Rohith Sharma K S
YARN-9821 NM hangs at serviceStop when ATSV2 Backend Hbase is Down Major ATSv2 Prabhu Joseph Prabhu Joseph

OTHER:

JIRA Summary Priority Component Reporter Contributor
HADOOP-15851 Disable wildfly logs to the console Major fs/azure Vishwajeet Dusane Vishwajeet Dusane
HDFS-12729 Document special paths in HDFS Major documentation Chris Douglas Masatake Iwasaki
YARN-9191 Add cli option in DS to support enforceExecutionType in resource requests. Major . Abhishek Modi Abhishek Modi
HADOOP-16037 DistCp: Document usage of Sync (-diff option) in detail Major documentation, tools/distcp Siyao Meng Siyao Meng
HADOOP-16263 Update BUILDING.txt with macOS native build instructions Minor . Siyao Meng Siyao Meng
YARN-9559 Create AbstractContainersLauncher for pluggable ContainersLauncher logic Major . Jonathan Hung Jonathan Hung
YARN-9796 Fix ASF license issue in branch-3.2 Blocker . Rohith Sharma K S Prabhu Joseph