Apache Hadoop Changelog

Release 3.1.2 - 2019-01-29

NEW FEATURES:

JIRA Summary Priority Component Reporter Contributor
HDFS-13448 HDFS Block Placement - Ignore Locality for First Block Replica Minor block placement, hdfs-client BELUGA BEHR BELUGA BEHR
HADOOP-15677 WASB: Add support for StreamCapabilities Major fs/azure Thomas Marquardt Thomas Marquardt
HADOOP-15996 Plugin interface to support more complex usernames in Hadoop Major security Eric Yang Bolke de Bruin

IMPROVEMENTS:

JIRA Summary Priority Component Reporter Contributor
YARN-8226 Improve anti-affinity section description in YARN Service API doc Major docs, documentation Charan Hebri Gour Saha
HADOOP-15609 Retry KMS calls when SSLHandshakeException occurs Major common, kms Kitti Nanasi Kitti Nanasi
HADOOP-15612 Improve exception when tfile fails to load LzoCodec Major . Gera Shegalov Gera Shegalov
HDFS-11060 make DEFAULT_MAX_CORRUPT_FILEBLOCKS_RETURNED configurable Minor hdfs Lantao Jin Lantao Jin
HDFS-13727 Log full stack trace if DiskBalancer exits with an unhandled exception Minor diskbalancer Stephen O’Donnell Gabor Bota
YARN-8584 Several typos in Log Aggregation related classes Minor . Szilard Nemeth Szilard Nemeth
HDFS-13728 Disk Balancer should not fail if volume usage is greater than capacity Minor diskbalancer Stephen O’Donnell Stephen O’Donnell
HDFS-13447 Fix Typos - Node Not Chosen Trivial namenode BELUGA BEHR BELUGA BEHR
YARN-8601 Print ExecutionType in Container report CLI Minor . Bilwa S T Bilwa S T
HDFS-13658 Expose HighestPriorityLowRedundancy blocks statistics Major hdfs Kitti Nanasi Kitti Nanasi
YARN-8568 Replace the deprecated zk-address property in the HA config example in ResourceManagerHA.md Minor yarn Antal Bálint Steinbach Antal Bálint Steinbach
HDFS-13735 Make QJM HTTP URL connection timeout configurable Minor qjm Chao Sun Chao Sun
HDFS-13814 Remove super user privilege requirement for NameNode.getServiceStatus Minor namenode Chao Sun Chao Sun
YARN-8559 Expose mutable-conf scheduler’s configuration in RM /scheduler-conf endpoint Major resourcemanager Anna Savarin Weiwei Yang
HDFS-13813 Exit NameNode if dangling child inode is detected when saving FsImage Major hdfs, namenode Siyao Meng Siyao Meng
HADOOP-14212 Expose SecurityEnabled boolean field in JMX for other services besides NameNode Minor . Ray Burgemeestre Adam Antal
HDFS-13217 Audit log all EC policy names during addErasureCodingPolicies Major erasure-coding liaoyuxiangqin liaoyuxiangqin
HADOOP-9214 Create a new touch command to allow modifying atime and mtime Minor tools Brian Burton Hrishikesh Gadre
YARN-8242 YARN NM: OOM error while reading back the state store on recovery Critical yarn Kanwaljeet Sachdev Pradeep Ambati
HDFS-13821 RBF: Add dfs.federation.router.mount-table.cache.enable so that users can disable cache Major hdfs Fei Hui Fei Hui
HDFS-13861 RBF: Illegal Router Admin command leads to printing usage for all commands Major . Ayush Saxena Ayush Saxena
HDFS-13831 Make block increment deletion number configurable Major . Yiqun Lin Ryan Wu
YARN-8051 TestRMEmbeddedElector#testCallbackSynchronization is flakey Major test Robert Kanter Robert Kanter
HDFS-13857 RBF: Choose to enable the default nameservice to read/write files Major federation, hdfs yanghuafeng yanghuafeng
HDFS-13812 Fix the inconsistent default refresh interval on Caching documentation Trivial documentation BELUGA BEHR Hrishikesh Gadre
YARN-8638 Allow linux container runtimes to be pluggable Minor nodemanager Craig Condit Craig Condit
HDFS-13902 Add JMX, conf and stacks menus to the datanode page Minor datanode fengchuang fengchuang
YARN-8680 YARN NM: Implement Iterable Abstraction for LocalResourceTracker state Critical yarn Pradeep Ambati Pradeep Ambati
HADOOP-15726 Create utility to limit frequency of log statements Major common, util Erik Krogen Erik Krogen
YARN-7974 Allow updating application tracking url after registration Major . Jonathan Hung Jonathan Hung
YARN-8758 Support getting PreemptionMessage when using AMRMClientAsync Major yarn Krishna Kishore Zian Chen
YARN-8896 Limit the maximum number of container assignments per heartbeat Major . Weiwei Yang Zhankun Tang
HADOOP-15804 upgrade to commons-compress 1.18 Major . PJ Fanning Akira Ajisaka
YARN-8908 Fix errors in yarn-default.xml related to GPU/FPGA Major . Zhankun Tang Zhankun Tang
HDFS-13941 make storageId in BlockPoolTokenSecretManager.checkAccess optional Major . Ajay Kumar Ajay Kumar
HDFS-14029 Sleep in TestLazyPersistFiles should be put into a loop Trivial hdfs Adam Antal Adam Antal
YARN-8915 Update the doc about the default value of “maximum-container-assignments” for capacity scheduler Minor . Zhankun Tang Zhankun Tang
HADOOP-15855 Review hadoop credential doc, including object store details Minor documentation, security Steve Loughran Steve Loughran
YARN-7225 Add queue and partition info to RM audit log Major resourcemanager Jonathan Hung Eric Payne
YARN-8969 AbstractYarnScheduler#getNodeTracker should return generic type to avoid type casting Major . Wanqiang Ji Wanqiang Ji
YARN-8977 Remove unnecessary type casting when calling AbstractYarnScheduler#getSchedulerNode Trivial . Wanqiang Ji Wanqiang Ji
HDFS-14070 Refactor NameNodeWebHdfsMethods to allow better extensibility Major . CR Hota CR Hota
HADOOP-12558 distcp documentation is woefully out of date Critical documentation, tools/distcp Allen Wittenauer Dinesh Chitlangia
HADOOP-15919 AliyunOSS: Enable Yarn to use OSS Major fs/oss wujinhu wujinhu
HADOOP-15943 AliyunOSS: add missing owner & group attributes for oss FileStatus Major fs/oss wujinhu wujinhu
MAPREDUCE-7164 FileOutputCommitter does not report progress while merging paths. Major . Kuhu Shukla Kuhu Shukla
YARN-9069 Fix SchedulerInfo#getSchedulerType for custom schedulers Minor . Bilwa S T Bilwa S T
HDFS-14095 EC: Track Erasure Coding commands in DFS statistics Major erasure-coding Ayush Saxena Ayush Saxena
YARN-9036 Escape newlines in health report in YARN UI Major . Jonathan Hung Keqiu Hu
YARN-9085 Add Guaranteed and MaxCapacity to CSQueueMetrics Major . Jonathan Hung Jonathan Hung
HADOOP-15808 Harden Token service loader use Major security Steve Loughran Steve Loughran
YARN-9122 Add table of contents to YARN Service API document Minor documentation Akira Ajisaka Zhankun Tang
HDFS-14171 Performance improvement in Tailing EditLog Major namenode Kenneth Yang Kenneth Yang
HADOOP-15959 revert HADOOP-12751 Minor security Steve Loughran Steve Loughran
HADOOP-16019 ZKDelegationTokenSecretManager won’t log exception message occured in function setJaasConfiguration Minor common luhuachao luhuachao

BUG FIXES:

JIRA Summary Priority Component Reporter Contributor
YARN-7773 YARN Federation used Mysql as state store throw exception, Unknown column ‘homeSubCluster’ in ‘field list’ Blocker federation Yiran Wu Yiran Wu
YARN-8426 Upgrade jquery-ui to 1.12.1 in YARN Major webapp Sunil Govindan Sunil Govindan
HADOOP-15550 Avoid static initialization of ObjectMappers Minor performance Todd Lipcon Todd Lipcon
HDFS-13721 NPE in DataNode due to uninitialized DiskBalancer Major datanode, diskbalancer Xiao Chen Xiao Chen
YARN-8360 Yarn service conflict between restart policy and NM configuration Critical yarn Chandni Singh Suma Shivaprasad
YARN-8380 Support bind propagation options for mounts in docker runtime Major . Billie Rinaldi Billie Rinaldi
YARN-8544 [DS] AM registration fails when hadoop authorization is enabled Blocker . Bibin A Chundatt Bibin A Chundatt
YARN-8548 AllocationRespose proto setNMToken initBuilder not done Major . Bibin A Chundatt Bilwa S T
YARN-7748 TestContainerResizing.testIncreaseContainerUnreservedWhenApplicationCompleted fails due to multiple container fail events Major capacityscheduler Haibo Chen Weiwei Yang
YARN-8577 Fix the broken anchor in SLS site-doc Minor documentation Weiwei Yang Weiwei Yang
YARN-4606 CapacityScheduler: applications could get starved because computation of #activeUsers considers pending apps Critical capacity scheduler, capacityscheduler Karam Singh Manikandan R
YARN-8330 Avoid publishing reserved container to ATS from RM Critical yarn-native-services Yesha Vora Suma Shivaprasad
YARN-8429 Improve diagnostic message when artifact is not set properly Major . Yesha Vora Gour Saha
YARN-8571 Validate service principal format prior to launching yarn service Major security, yarn Eric Yang Eric Yang
HADOOP-15637 LocalFs#listLocatedStatus does not filter out hidden .crc files Minor fs Erik Krogen Erik Krogen
YARN-8579 New AM attempt could not retrieve previous attempt component data Critical . Yesha Vora Gour Saha
YARN-8397 Potential thread leak in ActivitiesManager Major . Rohith Sharma K S Rohith Sharma K S
YARN-8595 [UI2] Container diagnostic information is missing from container page Major yarn-ui-v2 Akhil PB Akhil PB
YARN-8403 Nodemanager logs failed to download file with INFO level Major yarn Eric Yang Eric Yang
YARN-8610 Yarn Service Upgrade: Typo in Error message Major . Chandni Singh Chandni Singh
YARN-8593 Add RM web service endpoint to get user information Major resourcemanager Akhil PB Akhil PB
YARN-8594 [UI2] Display current logged in user Major . Akhil PB Akhil PB
YARN-8592 [UI2] rmip:port/ui2 endpoint shows a blank page in windows OS and Chrome browser Major . Akhil S Naik Akhil PB
YARN-8318 [UI2] IP address in component page shows N/A Major yarn-ui-v2 Yesha Vora Yesha Vora
YARN-6966 NodeManager metrics may return wrong negative values when NM restart Major . Yang Wang Szilard Nemeth
YARN-8620 [UI2] YARN Services UI new submission failures are not debuggable Major yarn-ui-v2 Akhil PB Akhil PB
YARN-8615 [UI2] Resource Usage tab shows only memory related info. No info available for vcores/gpu. Major yarn-ui-v2 Sumana Sathish Akhil PB
HDFS-13792 Fix FSN read/write lock metrics name Trivial documentation, metrics Chao Sun Chao Sun
YARN-8629 Container cleanup fails while trying to delete Cgroups Critical . Yesha Vora Suma Shivaprasad
YARN-8407 Container launch exception in AM log should be printed in ERROR level Major . Yesha Vora Yesha Vora
HDFS-13799 TestEditLogTailer#testTriggersLogRollsForAllStandbyNN fails due to missing synchronization between rollEditsRpcExecutor and tailerThread shutdown Minor ha Hrishikesh Gadre Hrishikesh Gadre
HDFS-13786 EC: Display erasure coding policy for sub-directories is not working Major erasure-coding Souryakanta Dwivedy Ayush Saxena
HDFS-13785 EC: “removePolicy” is not working for built-in/system Erasure Code policies Minor documentation, erasure-coding Souryakanta Dwivedy Ayush Saxena
YARN-8633 Update DataTables version in yarn-common in line with JQuery 3 upgrade Major yarn Akhil PB Akhil PB
YARN-8331 Race condition in NM container launched after done Major . Yang Wang Pradeep Ambati
YARN-8521 NPE in AllocationTagsManager when a container is removed more than once Major resourcemanager Weiwei Yang Weiwei Yang
YARN-8575 Avoid committing allocation proposal to unavailable nodes in async scheduling Major capacityscheduler Tao Yang Tao Yang
HDFS-13668 FSPermissionChecker may throws AIOOE when check inode permission Major namenode He Xiaoqiao He Xiaoqiao
HADOOP-15638 KMS Accept Queue Size default changed from 500 to 128 in Hadoop 3.x Major kms Wei-Chiu Chuang Wei-Chiu Chuang
HDFS-13823 NameNode UI : “Utilities -> Browse the file system -> open a file -> Head the file” is not working Major ui Nanda kumar Nanda kumar
HDFS-13738 fsck -list-corruptfileblocks has infinite loop if user is not privileged. Major tools Wei-Chiu Chuang Yuen-Kuei Hsueh
HDFS-13758 DatanodeManager should throw exception if it has BlockRecoveryCommand but the block is not under construction Major namenode Wei-Chiu Chuang chencan
YARN-8614 Fix few annotation typos in YarnConfiguration Trivial . Sen Zhao Sen Zhao
HDFS-13819 TestDirectoryScanner#testDirectoryScannerInFederatedCluster is flaky Minor hdfs Daniel Templeton Daniel Templeton
YARN-8656 container-executor should not write cgroup tasks files for docker containers Major . Jim Brennan Jim Brennan
YARN-8474 sleeper service fails to launch with “Authentication Required” Critical yarn Sumana Sathish Billie Rinaldi
YARN-8667 Cleanup symlinks when container restarted by NM to solve issue “find: File system loop detected;” for tar ball artifacts. Critical . Rohith Sharma K S Chandni Singh
HDFS-10240 Race between close/recoverLease leads to missing block Major . zhouyingchao Jinglun
HADOOP-15655 Enhance KMS client retry behavior Critical kms Kitti Nanasi Kitti Nanasi
YARN-8612 Fix NM Collector Service Port issue in YarnConfiguration Major ATSv2 Prabha Manepalli Prabha Manepalli
HDFS-13747 Statistic for list_located_status is incremented incorrectly by listStatusIterator Minor hdfs-client Todd Lipcon Antal Mihalyi
HADOOP-15674 Test failure TestSSLHttpServer.testExcludedCiphers with TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256 cipher suite Major common Gabor Bota Szilard Nemeth
YARN-8640 Restore previous state in container-executor after failure Major . Jim Brennan Jim Brennan
YARN-8679 [ATSv2] If HBase cluster is down for long time, high chances that NM ContainerManager dispatcher get blocked Major . Rohith Sharma K S Wangda Tan
HDFS-13772 Erasure coding: Unnecessary NameNode Logs displaying for Enabling/Disabling Erasure coding policies which are already enabled/disabled Trivial erasure-coding Souryakanta Dwivedy Ayush Saxena
YARN-8649 NPE in localizer hearbeat processing if a container is killed while localizing Major . lujie lujie
YARN-8719 Typo correction for yarn configuration in OpportunisticContainers(federation) docs Major documentation, federation Y. SREENIVASULU REDDY Y. SREENIVASULU REDDY
YARN-8675 Setting hostname of docker container breaks with “host” networking mode for Apps which do not run as a YARN service Major . Yesha Vora Suma Shivaprasad
HDFS-13858 RBF: Add check to have single valid argument to safemode command Major federation Soumyapn Ayush Saxena
HDFS-13731 ReencryptionUpdater fails with ConcurrentModificationException during processCheckpoints Major encryption Xiao Chen Zsolt Venczel
YARN-8723 Fix a typo in CS init error message when resource calculator is not correctly set Minor . Weiwei Yang Abhishek Modi
HADOOP-15705 Typo in the definition of “stable” in the interface classification Minor . Daniel Templeton Daniel Templeton
HDFS-13863 FsDatasetImpl should log DiskOutOfSpaceException Major hdfs Fei Hui Fei Hui
HADOOP-15698 KMS log4j is not initialized properly at startup Major kms Kitti Nanasi Kitti Nanasi
HADOOP-15680 ITestNativeAzureFileSystemConcurrencyLive times out Major . Andras Bokor Andras Bokor
HADOOP-15706 Typo in compatibility doc: SHOUD -> SHOULD Trivial . Daniel Templeton Laszlo Kollar
HDFS-13027 Handle possible NPEs due to deleted blocks in race condition Major namenode Vinayakumar B Vinayakumar B
YARN-8535 Fix DistributedShell unit tests Major distributed-shell, timelineservice Eric Yang Abhishek Modi
HDFS-13867 RBF: Add validation for max arguments for Router admin ls, clrQuota, setQuota, rm and nameservice commands Major . Ayush Saxena Ayush Saxena
HDFS-13774 EC: “hdfs ec -getPolicy” is not retrieving policy details when the special REPLICATION policy set on the directory Minor erasure-coding Souryakanta Dwivedy Ayush Saxena
HADOOP-10219 ipc.Client.setupIOstreams() needs to check for ClientCache.stopClient requested shutdowns Major ipc Steve Loughran Kihwal Lee
HDFS-13815 RBF: Add check to order command Major federation Soumyapn Ranith Sardar
HADOOP-15696 KMS performance regression due to too many open file descriptors after Jetty migration Blocker kms Wei-Chiu Chuang Wei-Chiu Chuang
MAPREDUCE-7131 Job History Server has race condition where it moves files from intermediate to finished but thinks file is in intermediate Major . Anthony Hsu Anthony Hsu
HDFS-13836 RBF: Handle mount table znode with null value Major federation, hdfs yanghuafeng yanghuafeng
YARN-8751 Container-executor permission check errors cause the NM to be marked unhealthy Critical . Shane Kumpf Craig Condit
HDFS-13862 RBF: Router logs are not capturing few of the dfsrouteradmin commands Major . Soumyapn Ayush Saxena
HDFS-12716 ‘dfs.datanode.failed.volumes.tolerated’ to support minimum number of volumes to be available Major datanode usharani Ranith Sardar
HDFS-13895 EC: Fix Intermittent Failure in TestDFSStripedOutputStreamWithFailureWithRandomECPolicy Major erasure-coding Ayush Saxena Ayush Saxena
YARN-8709 CS preemption monitor always fails since one under-served queue was deleted Major capacityscheduler, scheduler preemption Tao Yang Tao Yang
HDFS-13051 Fix dead lock during async editlog rolling if edit queue is full Major namenode zhangwei Daryn Sharp
YARN-8630 ATSv2 REST APIs should honor filter-entity-list-by-user in non-secure cluster when ACls are enabled Major . Rohith Sharma K S Rohith Sharma K S
YARN-8729 Node status updater thread could be lost after it is restarted Critical nodemanager Tao Yang Tao Yang
HDFS-13914 Fix DN UI logs link broken when https is enabled after HDFS-13902 Minor datanode Jianfei Jiang Jianfei Jiang
MAPREDUCE-7133 History Server task attempts REST API returns invalid data Major jobhistoryserver Oleksandr Shevchenko Oleksandr Shevchenko
HDFS-13838 WebHdfsFileSystem.getFileStatus() won’t return correct “snapshot enabled” status Major hdfs, webhdfs Siyao Meng Siyao Meng
HADOOP-15733 Correct the log when Invalid emptier Interval configured Major trash Harshakiran Reddy Ayush Saxena
YARN-8720 CapacityScheduler does not enforce max resource allocation check at queue level Major capacity scheduler, capacityscheduler, resourcemanager Tarun Parimi Tarun Parimi
YARN-8782 Fix exception message in Resource.throwExceptionWhenArrayOutOfBound Minor . Szilard Nemeth Gergely Pollak
HDFS-13844 Fix the fmt_bytes function in the dfs-dust.js Minor hdfs, ui yanghuafeng yanghuafeng
YARN-8726 [UI2] YARN UI2 is not accessible when config.env file failed to load Critical yarn-ui-v2 Akhil PB Akhil PB
YARN-8652 [UI2] YARN UI2 breaks if getUserInfo REST API is not available in older versions. Critical . Akhil PB Akhil PB
YARN-8787 Fix broken list items in PlacementConstraints documentation Minor documentation Masahiro Tanaka Masahiro Tanaka
HDFS-11719 Arrays.fill() wrong index in BlockSender.readChecksum() exception handling Major datanode Tao Zhang Tao Zhang
YARN-8648 Container cgroups are leaked when using docker Major . Jim Brennan Jim Brennan
HADOOP-15755 StringUtils#createStartupShutdownMessage throws NPE when args is null Major . Lokesh Jain Dinesh Chitlangia
MAPREDUCE-3801 org.apache.hadoop.mapreduce.v2.app.TestRuntimeEstimators.testExponentialEstimator fails intermittently Major mrv2 Robert Joseph Evans Jason Lowe
MAPREDUCE-7137 MRAppBenchmark.benchmark1() fails with NullPointerException Minor test Oleksandr Shevchenko Oleksandr Shevchenko
MAPREDUCE-7138 ThrottledContainerAllocator in MRAppBenchmark should implement RMHeartbeatHandler Minor test Oleksandr Shevchenko Oleksandr Shevchenko
HDFS-13886 HttpFSFileSystem.getFileStatus() doesn’t return “snapshot enabled” bit Major httpfs Siyao Meng Siyao Meng
HDFS-13868 WebHDFS: GETSNAPSHOTDIFF API NPE when param “snapshotname” is given but “oldsnapshotname” is not. Major hdfs, webhdfs Siyao Meng Pranay Singh
YARN-8771 CapacityScheduler fails to unreserve when cluster resource contains empty resource type Critical capacityscheduler Tao Yang Tao Yang
HDFS-13908 TestDataNodeMultipleRegistrations is flaky Major . Íñigo Goiri Ayush Saxena
HADOOP-15684 triggerActiveLogRoll stuck on dead name node, when ConnectTimeoutException happens. Critical ha Rong Tang Rong Tang
HADOOP-15772 Remove the ‘Path … should be specified as a URI’ warnings on startup Major conf Arpit Agarwal Ayush Saxena
YARN-8784 DockerLinuxContainerRuntime prevents access to distributed cache entries on a full disk Major nodemanager Jason Lowe Eric Badger
HADOOP-15736 Trash : Negative Value For Deletion Interval Leads To Abnormal Behaviour. Major . Ayush Saxena Ayush Saxena
YARN-8628 [UI2] Few duplicated or inconsistent information displayed in UI2 Major yarn-ui-v2 Akhil PB Akhil PB
YARN-8742 [UI2] Container logs on Application / Service pages on UI2 are not available many case, improve error messages in such cases. Major yarn-ui-v2 Akhil PB Akhil PB
MAPREDUCE-7125 JobResourceUploader creates LocalFileSystem when it’s not necessary Major job submission Peter Cseh Peter Cseh
YARN-8815 RM fails to recover finished unmanaged AM Critical . Rakesh Shah Bibin A Chundatt
YARN-8752 yarn-registry.md has wrong word ong-lived, it should be long-lived Trivial documentation leiqiang leiqiang
YARN-8824 App Nodelabel missed after RM restart for finished apps Major . Bibin A Chundatt Bibin A Chundatt
HDFS-13840 RBW Blocks which are having less GS should be added to Corrupt Minor . Brahma Reddy Battula Brahma Reddy Battula
YARN-8804 resourceLimits may be wrongly calculated when leaf-queue is blocked in cluster with 3+ level queues Critical capacityscheduler Tao Yang Tao Yang
YARN-8774 Memory leak when CapacityScheduler allocates from reserved container with non-default label Critical capacityscheduler Tao Yang Tao Yang
HADOOP-15817 Reuse Object Mapper in KMSJSONReader Major kms Jonathan Eagles Jonathan Eagles
YARN-8844 TestNMProxy unit test is failing Major yarn Eric Yang Eric Yang
HDFS-13957 Fix incorrect option used in description of InMemoryAliasMap Minor documentation Yiqun Lin Yiqun Lin
YARN-7957 [UI2] YARN service delete option disappears after stopping application Critical yarn-ui-v2 Yesha Vora Akhil PB
YARN-8797 [UI2] Improve error pages in new YARN UI Major yarn-ui-v2 Akhil PB Akhil PB
HADOOP-15820 ZStandardDecompressor native code sets an integer field as a long Blocker . Jason Lowe Jason Lowe
HDFS-13964 RBF: TestRouterWebHDFSContractAppend fails with No Active Namenode under nameservice Major . Ayush Saxena Ayush Saxena
YARN-7825 [UI2] Maintain constant horizontal application info bar for Application Attempt page Major yarn-ui-v2 Yesha Vora Akhil PB
HDFS-13768 Adding replicas to volume map makes DataNode start slowly Major . Yiqun Lin Surendra Singh Lilhore
HDFS-13962 Add null check for add-replica pool to avoid lock acquiring Major . Yiqun Lin Surendra Singh Lilhore
YARN-8853 [UI2] Application Attempts tab is not shown correctly when there are no attempts Major . Charan Hebri Akhil PB
YARN-8845 hadoop.registry.rm.enabled is not used Major . Íñigo Goiri Íñigo Goiri
HDFS-13926 ThreadLocal aggregations for FileSystem.Statistics are incorrect with striped reads Major erasure-coding Xiao Chen Hrishikesh Gadre
YARN-8666 [UI2] Remove application tab from YARN Queue Page Major yarn-ui-v2 Yesha Vora Yesha Vora
YARN-8753 [UI2] Lost nodes representation missing from Nodemanagers Chart Major yarn-ui-v2 Yesha Vora Yesha Vora
HADOOP-15679 ShutdownHookManager shutdown time needs to be configurable & extended Major util Steve Loughran Steve Loughran
HDFS-13945 TestDataNodeVolumeFailure is Flaky Major . Ayush Saxena Ayush Saxena
HDFS-13802 RBF: Remove FSCK from Router Web UI Major . Fei Hui Fei Hui
YARN-8869 YARN Service Client might not work correctly with RM REST API for Kerberos authentication Blocker . Eric Yang Eric Yang
MAPREDUCE-7132 JobSplitWriter prints unnecessary warnings if EC(RS10,4) is used Major client, mrv2 Peter Bacsko Peter Bacsko
YARN-8892 YARN UI2 doc changes to update security status (verified under security environment) Blocker . Sunil Govindan Sunil Govindan
YARN-8810 Yarn Service: discrepancy between hashcode and equals of ConfigFile Minor . Chandni Singh Chandni Singh
HADOOP-15802 start-build-env.sh creates an invalid /etc/sudoers.d/hadoop-build-${USER_ID} file entry Minor build Jon Boone Jon Boone
HADOOP-15861 Move DelegationTokenIssuer to the right path Blocker security Wei-Chiu Chuang Wei-Chiu Chuang
HADOOP-15859 ZStandardDecompressor.c mistakes a class for an instance Blocker . Ben Lau Jason Lowe
HDFS-14000 RBF: Documentation should reflect right scripts for v3.0 and above Major . CR Hota CR Hota
YARN-8868 Set HTTPOnly attribute to Cookie Major . Chandni Singh Chandni Singh
HDFS-14003 Fix findbugs warning in trunk for FSImageFormatPBINode Major . Yiqun Lin Yiqun Lin
HDFS-14005 RBF: Web UI update to bootstrap-3.3.7 Major . Íñigo Goiri Íñigo Goiri
YARN-8910 Misleading log statement in NM when max retries is -1 Minor . Chandni Singh Chandni Singh
HADOOP-15850 CopyCommitter#concatFileChunks should check that the blocks per chunk is not 0 Critical tools/distcp Ted Yu Ted Yu
YARN-7502 Nodemanager restart docs should describe nodemanager supervised property Major documentation Jason Lowe Suma Shivaprasad
HADOOP-15866 Renamed HADOOP_SECURITY_GROUP_SHELL_COMMAND_TIMEOUT keys break compatibility Blocker . Wei-Chiu Chuang Wei-Chiu Chuang
YARN-8826 Fix lingering timeline collector after serviceStop in TimelineCollectorManager Trivial ATSv2 Prabha Manepalli Prabha Manepalli
HADOOP-15822 zstd compressor can fail with a small output buffer Major . Jason Lowe Jason Lowe
HDFS-14021 TestReconstructStripedBlocksWithRackAwareness#testReconstructForNotEnoughRacks fails intermittently Major erasure-coding, test Xiao Chen Xiao Chen
MAPREDUCE-7151 RMContainerAllocator#handleJobPriorityChange expects application_priority always Major . Bibin A Chundatt Bilwa S T
HADOOP-14445 Use DelegationTokenIssuer to create KMS delegation tokens that can authenticate to all KMS instances Major kms Wei-Chiu Chuang Xiao Chen
HDFS-14028 HDFS OIV temporary dir deletes folder Major hdfs Adam Antal Adam Antal
HDFS-14027 DFSStripedOutputStream should implement both hsync methods Critical erasure-coding Xiao Chen Xiao Chen
YARN-8950 Compilation fails with dependency convergence error for hbase.profile=2.0 Blocker . Rohith Sharma K S Rohith Sharma K S
HADOOP-15899 Update AWS Java SDK versions in NOTICE.txt Major . Akira Ajisaka Akira Ajisaka
HADOOP-15900 Update JSch versions in LICENSE.txt Major . Akira Ajisaka Akira Ajisaka
HDFS-14043 Tolerate corrupted seen_txid file Major hdfs, namenode Lukas Majercak Lukas Majercak
YARN-8858 CapacityScheduler should respect maximum node resource when per-queue maximum-allocation is being used. Major . Sumana Sathish Wangda Tan
YARN-8970 Improve the debug message in CS#allocateContainerOnSingleNode Trivial . Weiwei Yang Zhankun Tang
YARN-8865 RMStateStore contains large number of expired RMDelegationToken Major resourcemanager Wilfred Spiegelenburg Wilfred Spiegelenburg
HDFS-14048 DFSOutputStream close() throws exception on subsequent call after DataNode restart Major hdfs-client Erik Krogen Erik Krogen
MAPREDUCE-7156 NullPointerException when reaching max shuffle connections Major mrv2 Peter Bacsko Peter Bacsko
YARN-8866 Fix a parsing error for crossdomain.xml Major build, yarn-ui-v2 Takanobu Asanuma Takanobu Asanuma
HDFS-14039 ec -listPolicies doesn’t show correct state for the default policy when the default is not RS(6,3) Major erasure-coding Xiao Chen Kitti Nanasi
HADOOP-15916 Upgrade Maven Surefire plugin to 3.0.0-M1 Blocker build Akira Ajisaka Akira Ajisaka
YARN-9002 YARN Service keytab does not support s3, wasb, gs and is restricted to HDFS and local filesystem only Major yarn-native-services Gour Saha Gour Saha
YARN-8233 NPE in CapacityScheduler#tryCommit when handling allocate/reserve proposal whose allocatedOrReservedContainer is null Critical capacityscheduler Tao Yang Tao Yang
HDFS-14065 Failed Storage Locations shows nothing in the Datanode Volume Failures Major . Ayush Saxena Ayush Saxena
HADOOP-15923 create-release script should set max-cache-ttl as well as default-cache-ttl for gpg-agent Blocker build Akira Ajisaka Akira Ajisaka
HADOOP-15912 start-build-env.sh still creates an invalid /etc/sudoers.d/hadoop-build-${USER_ID} file entry after HADOOP-15802 Major build Akira Ajisaka Akira Ajisaka
HADOOP-15869 BlockDecompressorStream#decompress should not return -1 in case of IOException. Major . Surendra Singh Lilhore Surendra Singh Lilhore
MAPREDUCE-7158 Inefficient Flush Logic in JobHistory EventWriter Major . Zichen Sun Zichen Sun
HADOOP-15930 Exclude MD5 checksum files from release artifact Critical build Akira Ajisaka Akira Ajisaka
HADOOP-15925 The config and log of gpg-agent are removed in create-release script Major build Akira Ajisaka Dinesh Chitlangia
HDFS-13963 NN UI is broken with IE11 Minor namenode, ui Daisuke Kobayashi Ayush Saxena
HDFS-14056 Fix error messages in HDFS-12716 Minor hdfs Adam Antal Ayush Saxena
MAPREDUCE-7162 TestEvents#testEvents fails Critical jobhistoryserver, test Zhaohui Xin Zhaohui Xin
YARN-9056 Yarn Service Upgrade: Instance state changes from UPGRADING to READY without performing a readiness check Critical . Chandni Singh Chandni Singh
YARN-8812 Containers fail during creating a symlink which started with hyphen for a resource file Minor . Oleksandr Shevchenko Oleksandr Shevchenko
YARN-9067 YARN Resource Manager is running OOM because of leak of Configuration Object Major yarn-native-services Eric Yang Eric Yang
MAPREDUCE-7165 mapred-site.xml is misformatted in single node setup document Major documentation Akira Ajisaka Zhaohui Xin
HADOOP-15970 Upgrade plexus-utils from 2.0.5 to 3.1.0 Major security Akira Ajisaka Akira Ajisaka
HADOOP-15974 Upgrade Curator version to 2.13.0 to fix ZK tests Major . Jason Lowe Akira Ajisaka
YARN-9071 NM and service AM don’t have updated status for reinitialized containers Critical . Billie Rinaldi Chandni Singh
MAPREDUCE-7159 FrameworkUploader: ensure proper permissions of generated framework tar.gz if restrictive umask is used Major mrv2 Peter Bacsko Peter Bacsko
YARN-9009 Fix flaky test TestEntityGroupFSTimelineStore.testCleanLogs Minor . OrDTesters OrDTesters
MAPREDUCE-7170 Doc typo in PluggableShuffleAndPluggableSort.md Minor documentation Zhaohui Xin Zhaohui Xin
YARN-9040 LevelDBCacheTimelineStore in ATS 1.5 leaks native memory Major timelineserver Tarun Parimi Tarun Parimi
YARN-9084 Service Upgrade: With default readiness check, the status of upgrade is reported to be successful prematurely Major . Chandni Singh Chandni Singh
HDFS-13661 Ls command with e option fails when the filesystem is not HDFS Major erasure-coding, tools Takanobu Asanuma Takanobu Asanuma
HADOOP-15973 Configuration: Included properties are not cached if resource is a stream Critical . Eric Payne Eric Payne
YARN-9154 Fix itemization in YARN service quickstart document Minor documentation Akira Ajisaka Ayush Saxena
HDFS-14166 Ls with -e option not giving the result in proper format Major . Soumyapn Shubham Dewan
HDFS-14046 In-Maintenance ICON is missing in datanode info page Major datanode Harshakiran Reddy Ranith Sardar
MAPREDUCE-7174 MapReduce example wordmedian should handle generic options Major . Fei Hui Fei Hui
YARN-9164 Shutdown NM may cause NPE when opportunistic container scheduling is enabled Critical . lujie lujie
HADOOP-15997 KMS client uses wrong UGI after HADOOP-14445 Blocker kms Wei-Chiu Chuang Wei-Chiu Chuang
HADOOP-15992 JSON License is included in the transitive dependency of aliyun-sdk-oss 3.0.0 Blocker . Akira Ajisaka Akira Ajisaka
HADOOP-16030 AliyunOSS: bring fixes back from HADOOP-15671 Blocker fs/oss wujinhu wujinhu
YARN-9173 FairShare calculation broken for large values after YARN-8833 Major fairscheduler Wilfred Spiegelenburg Wilfred Spiegelenburg
YARN-8833 Avoid potential integer overflow when computing fair shares Major fairscheduler liyakun liyakun
YARN-8747 [UI2] YARN UI2 page loading failed due to js error under some time zone configuration Critical webapp collinma collinma
YARN-9194 Invalid event: REGISTERED and LAUNCH_FAILED at FAILED, and NullPointerException happens in RM while shutdown a NM Critical . lujie lujie
YARN-9204 RM fails to start if absolute resource is specified for partition capacity in CS queues Blocker yarn Jiandan Yang Jiandan Yang
YARN-9210 RM nodes web page can not display node info Blocker yarn Jiandan Yang Jiandan Yang
YARN-9205 When using custom resource type, application will fail to run due to the CapacityScheduler throws InvalidResourceRequestException(GREATER_THEN_MAX_ALLOCATION) Critical . Zhankun Tang Zhankun Tang
HADOOP-15781 S3A assumed role tests failing due to changed error text in AWS exceptions Major fs/s3, test Steve Loughran Steve Loughran
HDFS-14228 Incorrect getSnapshottableDirListing() javadoc Major snapshots Wei-Chiu Chuang Dinesh Chitlangia

TESTS:

JIRA Summary Priority Component Reporter Contributor
YARN-8907 Modify a logging message in TestCapacityScheduler Trivial . Zhankun Tang Zhankun Tang
YARN-8904 TestRMDelegationTokens can fail in testRMDTMasterKeyStateOnRollingMasterKey Minor test Wilfred Spiegelenburg Wilfred Spiegelenburg
YARN-8944 TestContainerAllocation.testUserLimitAllocationMultipleContainers failure after YARN-8896 Minor capacity scheduler Wilfred Spiegelenburg Wilfred Spiegelenburg

SUB-TASKS:

JIRA Summary Priority Component Reporter Contributor
HDFS-13743 RBF: Router throws NullPointerException due to the invalid initialization of MountTableResolver Major . Takanobu Asanuma Takanobu Asanuma
HDFS-13583 RBF: Router admin clrQuota is not synchronized with nameservice Major . Dibyendu Karmakar Dibyendu Karmakar
YARN-8263 DockerClient still touches hadoop.tmp.dir Minor . Jason Lowe Craig Condit
YARN-8287 Update documentation and yarn-default related to the Docker runtime Minor . Shane Kumpf Craig Condit
YARN-8624 Cleanup ENTRYPOINT documentation Minor . Craig Condit Craig Condit
YARN-8136 Add version attribute to site doc examples and quickstart Major site Gour Saha Eric Yang
YARN-8588 Logging improvements for better debuggability Major . Suma Shivaprasad Suma Shivaprasad
YARN-8520 Document best practice for user management Major documentation, yarn Eric Yang Eric Yang
HDFS-13750 RBF: Router ID in RouterRpcClient is always null Major . Takanobu Asanuma Takanobu Asanuma
YARN-8129 Improve error message for invalid value in fields attribute Minor ATSv2 Charan Hebri Abhishek Modi
HDFS-13848 Refactor NameNode failover proxy providers Major ha, hdfs-client Konstantin Shvachko Konstantin Shvachko
HDFS-13634 RBF: Configurable value in xml for async connection request queue size. Major federation CR Hota CR Hota
YARN-8642 Add support for tmpfs mounts with the Docker runtime Major . Shane Kumpf Craig Condit
HADOOP-15107 Stabilize/tune S3A committers; review correctness & docs Blocker fs/s3 Steve Loughran Steve Loughran
HADOOP-15731 TestDistributedShell fails on Windows Major . Botong Huang Botong Huang
HDFS-13237 [Documentation] RBF: Mount points across multiple subclusters Minor . Íñigo Goiri Íñigo Goiri
HADOOP-15759 AliyunOSS: update oss-sdk version to 3.0.0 Major fs/oss wujinhu wujinhu
HADOOP-15748 S3 listing inconsistency can raise NPE in globber Major fs Steve Loughran Steve Loughran
YARN-8801 java doc comments in docker-util.h is confusing Minor . Zian Chen Zian Chen
HADOOP-15671 AliyunOSS: Support Assume Roles in AliyunOSS Major fs/oss wujinhu wujinhu
YARN-8623 Update Docker examples to use image which exists Minor . Craig Condit Craig Condit
HDFS-13790 RBF: Move ClientProtocol APIs to its own module Major . Íñigo Goiri Chao Sun
YARN-8785 Improve the error message when a bind mount is not whitelisted Major . Simon Prewo Simon Prewo
YARN-6989 Ensure timeline service v2 codebase gets UGI from HttpServletRequest in a consistent way Major timelineserver Vrushali C Abhishek Modi
YARN-5742 Serve aggregated logs of historical apps from timeline service Critical timelineserver Varun Saxena Rohith Sharma K S
YARN-8834 Provide Java client for fetching Yarn specific entities from TimelineReader Critical timelinereader Rohith Sharma K S Abhishek Modi
HADOOP-15837 DynamoDB table Update can fail S3A FS init Major fs/s3 Steve Loughran Steve Loughran
HADOOP-15839 Review + update cloud store sensitive keys in hadoop.security.sensitive-config-keys Minor conf Steve Loughran Steve Loughran
YARN-8687 YARN service example is out-dated Major yarn-native-services Eric Yang Suma Shivaprasad
YARN-6098 Add documentation for Delete Queue Major capacity scheduler, documentation Naganarasimha G R Suma Shivaprasad
HADOOP-15607 AliyunOSS: fix duplicated partNumber issue in AliyunOSSBlockOutputStream Critical . wujinhu wujinhu
HADOOP-15868 AliyunOSS: update document for properties of multiple part download, multiple part upload and directory copy Major fs/oss wujinhu wujinhu
HADOOP-15110 Gauges are getting logged in exceptions from AutoRenewalThreadForUserCreds Minor metrics, security Harshakiran Reddy LiXin Ge
HADOOP-15917 AliyunOSS: fix incorrect ReadOps and WriteOps in statistics Major fs/oss wujinhu wujinhu
YARN-8303 YarnClient should contact TimelineReader for application/attempt/container report Critical . Rohith Sharma K S Abhishek Modi
YARN-8299 Yarn Service Upgrade: Add GET APIs that returns instances matching query params Critical . Chandni Singh Chandni Singh
YARN-8160 Yarn Service Upgrade: Support upgrade of service that use docker containers Major . Chandni Singh Chandni Singh
YARN-8298 Yarn Service Upgrade: Support express upgrade of a service Major . Chandni Singh Chandni Singh
YARN-8986 publish all exposed ports to random ports when using bridge network Minor yarn Charo Zhang Charo Zhang
HADOOP-15932 Oozie unable to create sharelib in s3a filesystem Critical fs, fs/s3 Soumitra Sulav Steve Loughran
YARN-8665 Yarn Service Upgrade: Support cancelling upgrade Major . Chandni Singh Chandni Singh
HADOOP-16009 Replace the url of the repository in Apache Hadoop source code Major documentation Akira Ajisaka Akira Ajisaka
HADOOP-15323 AliyunOSS: Improve copy file performance for AliyunOSSFileSystemStore Major fs/oss wujinhu wujinhu
YARN-8822 Nvidia-docker v2 support for YARN GPU feature Critical . Zhankun Tang Charo Zhang

OTHER:

JIRA Summary Priority Component Reporter Contributor
YARN-8545 YARN native service should return container if launch failed Critical . Wangda Tan Chandni Singh
HDFS-13788 Update EC documentation about rack fault tolerance Major documentation, erasure-coding Xiao Chen Kitti Nanasi
HADOOP-15816 Upgrade Apache Zookeeper version due to security concerns Major . Boris Vulikh Akira Ajisaka
HADOOP-15882 Upgrade maven-shade-plugin from 2.4.3 to 3.2.0 Major . Takanobu Asanuma Takanobu Asanuma
HADOOP-15815 Upgrade Eclipse Jetty version to 9.3.24 Major . Boris Vulikh Boris Vulikh
YARN-8488 YARN service/components/instances should have SUCCEEDED/FAILED states Major yarn-native-services Wangda Tan Suma Shivaprasad