Apache Hadoop Changelog

Release 3.0.4 - Unreleased (as of 2018-09-02)

NEW FEATURES:

JIRA Summary Priority Component Reporter Contributor
HDFS-13448 HDFS Block Placement - Ignore Locality for First Block Replica Minor block placement, hdfs-client BELUGA BEHR BELUGA BEHR

IMPROVEMENTS:

JIRA Summary Priority Component Reporter Contributor
HADOOP-15252 Checkstyle version is not compatible with IDEA’s checkstyle plugin Major . Andras Bokor Andras Bokor
HDFS-13602 Add checkOperation(WRITE) checks in FSNamesystem Major ha, namenode Erik Krogen Chao Sun
HDFS-13155 BlockPlacementPolicyDefault.chooseTargetInOrder Not Checking Return Value for NULL Minor namenode BELUGA BEHR Zsolt Venczel
HDFS-13659 Add more test coverage for contentSummary for snapshottable path Major namenode, test Wei-Chiu Chuang Wei-Chiu Chuang
HADOOP-15499 Performance severe drop when running RawErasureCoderBenchmark with NativeRSRawErasureCoder Major . Sammi Chen Sammi Chen
HDFS-13653 Make dfs.client.failover.random.order a per nameservice configuration Major federation Ekanth Sethuramalingam Ekanth Sethuramalingam
YARN-8394 Improve data locality documentation for Capacity Scheduler Major . Weiwei Yang Weiwei Yang
HDFS-13641 Add metrics for edit log tailing Major metrics Chao Sun Chao Sun
HDFS-13686 Add overall metrics for FSNamesystemLock Major hdfs, namenode Lukas Majercak Lukas Majercak
HDFS-13692 StorageInfoDefragmenter floods log when compacting StorageInfo TreeSet Minor . Yiqun Lin Bharat Viswanadham
HDFS-13703 Avoid allocation of CorruptedBlocks hashmap when no corrupted blocks are hit Major performance Todd Lipcon Todd Lipcon
HADOOP-15554 Improve JIT performance for Configuration parsing Minor conf, performance Todd Lipcon Todd Lipcon
HDFS-13714 Fix TestNameNodePrunesMissingStorages test failures on Windows Major hdfs, namenode, test Lukas Majercak Lukas Majercak
HDFS-13712 BlockReaderRemote.read() logging improvement Minor hdfs-client Gergo Repas Gergo Repas
HDFS-13719 Docs around dfs.image.transfer.timeout are misleading Major . Kitti Nanasi Kitti Nanasi
HADOOP-15598 DataChecksum calculate checksum is contented on hashtable synchronization Major common Prasanth Jayachandran Prasanth Jayachandran
HADOOP-15609 Retry KMS calls when SSLHandshakeException occurs Major common, kms Kitti Nanasi Kitti Nanasi
HADOOP-15612 Improve exception when tfile fails to load LzoCodec Major . Gera Shegalov Gera Shegalov
HDFS-11060 make DEFAULT_MAX_CORRUPT_FILEBLOCKS_RETURNED configurable Minor hdfs Lantao Jin Lantao Jin
HDFS-13727 Log full stack trace if DiskBalancer exits with an unhandled exception Minor diskbalancer Stephen O’Donnell Gabor Bota
YARN-8155 Improve ATSv2 client logging in RM and NM publisher Major . Rohith Sharma K S Abhishek Modi
HDFS-13728 Disk Balancer should not fail if volume usage is greater than capacity Minor diskbalancer Stephen O’Donnell Stephen O’Donnell
YARN-8568 Replace the deprecated zk-address property in the HA config example in ResourceManagerHA.md Minor yarn Antal Bálint Steinbach Antal Bálint Steinbach
HDFS-13814 Remove super user privilege requirement for NameNode.getServiceStatus Minor namenode Chao Sun Chao Sun
YARN-8559 Expose mutable-conf scheduler’s configuration in RM /scheduler-conf endpoint Major resourcemanager Anna Savarin Weiwei Yang
HDFS-13813 Exit NameNode if dangling child inode is detected when saving FsImage Major hdfs, namenode Siyao Meng Siyao Meng
HDFS-13821 RBF: Add dfs.federation.router.mount-table.cache.enable so that users can disable cache Major hdfs Fei Hui Fei Hui
HDFS-13831 Make block increment deletion number configurable Major . Yiqun Lin Ryan Wu
YARN-8051 TestRMEmbeddedElector#testCallbackSynchronization is flakey Major test Robert Kanter Robert Kanter

BUG FIXES:

JIRA Summary Priority Component Reporter Contributor
HDFS-12857 StoragePolicyAdmin should support schema based path Major namenode Surendra Singh Lilhore Surendra Singh Lilhore
YARN-7835 [Atsv2] Race condition in NM while publishing events if second attempt is launched on the same node Critical . Rohith Sharma K S Rohith Sharma K S
YARN-7773 YARN Federation used Mysql as state store throw exception, Unknown column ‘homeSubCluster’ in ‘field list’ Blocker federation Yiran Wu Yiran Wu
HDFS-13636 Cross-Site Scripting vulnerability in HttpServer2 Major . Haibo Yan Haibo Yan
HDFS-13339 Volume reference can’t be released and may lead to deadlock when DataXceiver does a check volume Critical datanode liaoyuxiangqin Zsolt Venczel
YARN-8382 cgroup file leak in NM Major nodemanager Hu Ziqian Hu Ziqian
HDFS-13545 “guarded” is misspelled as “gaurded” in FSPermissionChecker.java Trivial documentation Jianchao Jia Jianchao Jia
MAPREDUCE-7103 Fix TestHistoryViewerPrinter on windows due to a mismatch line separator Minor . Giovanni Matteo Fumarola Giovanni Matteo Fumarola
HADOOP-15217 FsUrlConnection does not handle paths with spaces Major fs Joseph Fourny Zsolt Venczel
HDFS-12950 [oiv] ls will fail in secure cluster Major . Brahma Reddy Battula Wei-Chiu Chuang
YARN-8359 Exclude containermanager.linux test classes on Windows Major . Giovanni Matteo Fumarola Jason Lowe
HDFS-13664 Refactor ConfiguredFailoverProxyProvider to make inheritance easier Minor hdfs-client Chao Sun Chao Sun
HDFS-12670 can’t renew HDFS tokens with only the hdfs client jar Critical . Thomas Graves Arpit Agarwal
HDFS-13667 Typo: Marking all “datandoes” as stale Trivial namenode Wei-Chiu Chuang Nanda kumar
YARN-8405 RM zk-state-store.parent-path ACLs has been changed since HADOOP-14773 Major . Rohith Sharma K S Íñigo Goiri
YARN-8404 Timeline event publish need to be async to avoid Dispatcher thread leak in case ATS is down Blocker . Rohith Sharma K S Rohith Sharma K S
HDFS-13673 TestNameNodeMetrics fails on Windows Minor test Zuoming Zhang Zuoming Zhang
HDFS-13676 TestEditLogRace fails on Windows Minor test Zuoming Zhang Zuoming Zhang
HDFS-13174 hdfs mover -p /path times out after 20 min Major balancer & mover Istvan Fajth Istvan Fajth
HADOOP-15523 Shell command timeout given is in seconds whereas it is taken as millisec while scheduling Major . Bilwa S T Bilwa S T
HDFS-13682 Cannot create encryption zone after KMS auth token expires Critical encryption, kms, namenode Xiao Chen Xiao Chen
YARN-8444 NodeResourceMonitor crashes on bad swapFree value Major . Jim Brennan Jim Brennan
YARN-8443 Total #VCores in cluster metrics is wrong when CapacityScheduler reserved some containers Major webapp Tao Yang Tao Yang
YARN-8457 Compilation is broken with -Pyarn-ui Major webapp Sunil Govindan Sunil Govindan
YARN-8401 [UI2] new ui is not accessible with out internet connection Blocker . Bibin A Chundatt Bibin A Chundatt
YARN-8451 Multiple NM heartbeat thread created when a slow NM resync with RM Major nodemanager Botong Huang Botong Huang
HADOOP-15548 Randomize local dirs Minor . Jim Brennan Jim Brennan
HDFS-13702 Remove HTrace hooks from DFSClient to reduce CPU usage Major performance Todd Lipcon Todd Lipcon
HDFS-13635 Incorrect message when block is not found Major datanode Wei-Chiu Chuang Gabor Bota
HADOOP-15571 Multiple FileContexts created with the same configuration object should be allowed to have different umask Critical . Vinod Kumar Vavilapalli Vinod Kumar Vavilapalli
HDFS-13121 NPE when request file descriptors when SC read Minor hdfs-client Gang Xie Zsolt Venczel
YARN-6265 yarn.resourcemanager.fail-fast is used inconsistently Major resourcemanager Daniel Templeton Yuanbo Liu
YARN-8473 Containers being launched as app tears down can leave containers in NEW state Major nodemanager Jason Lowe Jason Lowe
HDFS-13723 Occasional “Should be different group” error in TestRefreshUserMappings#testGroupMappingRefresh Major security, test Siyao Meng Siyao Meng
HDFS-12837 Intermittent failure in TestReencryptionWithKMS Major encryption, test Surendra Singh Lilhore Xiao Chen
HDFS-13729 Fix broken links to RBF documentation Minor documentation jwhitter Gabor Bota
YARN-8515 container-executor can crash with SIGPIPE after nodemanager restart Major . Jim Brennan Jim Brennan
YARN-8421 when moving app, activeUsers is increased, even though app does not have outstanding request Major . kyungwan nam
HDFS-13524 Occasional “All datanodes are bad” error in TestLargeBlock#testLargeBlockSize Major . Wei-Chiu Chuang Siyao Meng
HADOOP-15610 Hadoop Docker Image Pip Install Fails Critical . Jack Bearden Jack Bearden
HADOOP-15614 TestGroupsCaching.testExceptionOnBackgroundRefreshHandled reliably fails Major . Kihwal Lee Weiwei Yang
YARN-8548 AllocationRespose proto setNMToken initBuilder not done Major . Bibin A Chundatt Bilwa S T
YARN-7748 TestContainerResizing.testIncreaseContainerUnreservedWhenApplicationCompleted fails due to multiple container fail events Major capacityscheduler Haibo Chen Weiwei Yang
YARN-8577 Fix the broken anchor in SLS site-doc Minor documentation Weiwei Yang Weiwei Yang
YARN-4606 CapacityScheduler: applications could get starved because computation of #activeUsers considers pending apps Critical capacity scheduler, capacityscheduler Karam Singh Manikandan R
HDFS-13765 Fix javadoc for FSDirMkdirOp#createParentDirectories Minor documentation Lokesh Jain Lokesh Jain
YARN-8434 Update federation documentation of Nodemanager configurations Minor . Bibin A Chundatt Bibin A Chundatt
YARN-8558 NM recovery level db not cleaned up properly on container finish Critical . Bibin A Chundatt Bibin A Chundatt
HADOOP-15637 LocalFs#listLocatedStatus does not filter out hidden .crc files Minor fs Erik Krogen Erik Krogen
YARN-8397 Potential thread leak in ActivitiesManager Major . Rohith Sharma K S Rohith Sharma K S
YARN-6966 NodeManager metrics may return wrong negative values when NM restart Major . Yang Wang Szilard Nemeth
HDFS-13786 EC: Display erasure coding policy for sub-directories is not working Major erasure-coding Souryakanta Dwivedy Ayush Saxena
YARN-8331 Race condition in NM container launched after done Major . Yang Wang Pradeep Ambati
HADOOP-15638 KMS Accept Queue Size default changed from 500 to 128 in Hadoop 3.x Major kms Wei-Chiu Chuang Wei-Chiu Chuang
HDFS-13738 fsck -list-corruptfileblocks has infinite loop if user is not privileged. Major tools Wei-Chiu Chuang Yuen-Kuei Hsueh
HDFS-13758 DatanodeManager should throw exception if it has BlockRecoveryCommand but the block is not under construction Major namenode Wei-Chiu Chuang chencan
YARN-8614 Fix few annotation typos in YarnConfiguration Trivial . Sen Zhao Sen Zhao
HDFS-13746 Still occasional “Should be different group” failure in TestRefreshUserMappings#testGroupMappingRefresh Major . Siyao Meng Siyao Meng
HDFS-10240 Race between close/recoverLease leads to missing block Major . zhouyingchao Jinglun
YARN-8612 Fix NM Collector Service Port issue in YarnConfiguration Major ATSv2 Prabha Manepalli Prabha Manepalli
HDFS-13747 Statistic for list_located_status is incremented incorrectly by listStatusIterator Minor hdfs-client Todd Lipcon Antal Mihalyi
HADOOP-15674 Test failure TestSSLHttpServer.testExcludedCiphers with TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256 cipher suite Major common Gabor Bota Szilard Nemeth
YARN-8640 Restore previous state in container-executor after failure Major . Jim Brennan Jim Brennan
YARN-8679 [ATSv2] If HBase cluster is down for long time, high chances that NM ContainerManager dispatcher get blocked Major . Rohith Sharma K S Wangda Tan
HADOOP-14314 The OpenSolaris taxonomy link is dead in InterfaceClassification.md Major documentation Daniel Templeton Rui Gao
YARN-8649 NPE in localizer hearbeat processing if a container is killed while localizing Major . lujie lujie
YARN-8719 Typo correction for yarn configuration in OpportunisticContainers(federation) docs Major documentation, federation Y. SREENIVASULU REDDY Y. SREENIVASULU REDDY
HDFS-13731 ReencryptionUpdater fails with ConcurrentModificationException during processCheckpoints Major encryption Xiao Chen Zsolt Venczel
HADOOP-15705 Typo in the definition of “stable” in the interface classification Minor . Daniel Templeton Daniel Templeton
HDFS-13863 FsDatasetImpl should log DiskOutOfSpaceException Major hdfs Fei Hui Fei Hui
HADOOP-15698 KMS log4j is not initialized properly at startup Major kms Kitti Nanasi Kitti Nanasi
HADOOP-15706 Typo in compatibility doc: SHOUD -> SHOULD Trivial . Daniel Templeton Laszlo Kollar
HDFS-13027 Handle possible NPEs due to deleted blocks in race condition Major namenode Vinayakumar B Vinayakumar B

TESTS:

JIRA Summary Priority Component Reporter Contributor
HDFS-13632 Randomize baseDir for MiniJournalCluster in MiniQJMHACluster for TestDFSAdminWithHA Minor . Anbang Hu Anbang Hu
HDFS-13651 TestReencryptionHandler fails on Windows Minor . Anbang Hu Anbang Hu
MAPREDUCE-7102 Fix TestJavaSerialization for Windows due a mismatch line separator Minor . Giovanni Matteo Fumarola Giovanni Matteo Fumarola
MAPREDUCE-7105 Fix TestNativeCollectorOnlyHandler.testOnCall on Windows because of the path format Minor . Giovanni Matteo Fumarola Giovanni Matteo Fumarola
HDFS-13652 Randomize baseDir for MiniDFSCluster in TestBlockScanner Minor . Anbang Hu Anbang Hu
HDFS-13649 Randomize baseDir for MiniDFSCluster in TestReconstructStripedFile and TestReconstructStripedFileWithRandomECPolicy Minor . Anbang Hu Anbang Hu
HDFS-13650 Randomize baseDir for MiniDFSCluster in TestDFSStripedInputStream and TestDFSStripedInputStreamWithRandomECPolicy Minor . Anbang Hu Anbang Hu
YARN-8370 Some Node Manager tests fail on Windows due to improper path/file separator Minor . Anbang Hu Anbang Hu
YARN-8422 TestAMSimulator failing with NPE Minor . Giovanni Matteo Fumarola Giovanni Matteo Fumarola
HADOOP-15532 TestBasicDiskValidator fails with NoSuchFileException Minor . Íñigo Goiri Giovanni Matteo Fumarola
HDFS-13563 TestDFSAdminWithHA times out on Windows Minor . Anbang Hu Lukas Majercak
HDFS-13681 Fix TestStartup.testNNFailToStartOnReadOnlyNNDir test failure on Windows Major test Xiao Liang Xiao Liang

SUB-TASKS:

JIRA Summary Priority Component Reporter Contributor
HDFS-12978 Fine-grained locking while consuming journal stream. Major namenode Konstantin Shvachko Konstantin Shvachko
HDFS-13637 RBF: Router fails when threadIndex (in ConnectionPool) wraps around Integer.MIN_VALUE Critical federation CR Hota CR Hota
HDFS-13281 Namenode#createFile should be /.reserved/raw/ aware. Critical encryption Rushabh S Shah Rushabh S Shah
YARN-4677 RMNodeResourceUpdateEvent update from scheduler can lead to race condition Major graceful, resourcemanager, scheduler Brook Zhou Wilfred Spiegelenburg
HADOOP-15506 Upgrade Azure Storage Sdk version to 7.0.0 and update corresponding code blocks Minor fs/azure Esfandiar Manii Esfandiar Manii
HADOOP-15529 ContainerLaunch#testInvalidEnvVariableSubstitutionType is not supported in Windows Minor . Giovanni Matteo Fumarola Giovanni Matteo Fumarola
HADOOP-15533 Make WASB listStatus messages consistent Trivial fs/azure Esfandiar Manii Esfandiar Manii
HADOOP-15458 TestLocalFileSystem#testFSOutputStreamBuilder fails on Windows Minor test Xiao Liang Xiao Liang
HDFS-13726 RBF: Fix RBF configuration links Minor documentation Takanobu Asanuma Takanobu Asanuma
HDFS-13475 RBF: Admin cannot enforce Router enter SafeMode Major . Wei Yan Chao Sun
HDFS-13733 RBF: Add Web UI configurations and descriptions to RBF document Minor documentation Takanobu Asanuma Takanobu Asanuma
HDFS-13743 RBF: Router throws NullPointerException due to the invalid initialization of MountTableResolver Major . Takanobu Asanuma Takanobu Asanuma
HDFS-13750 RBF: Router ID in RouterRpcClient is always null Major . Takanobu Asanuma Takanobu Asanuma
YARN-8129 Improve error message for invalid value in fields attribute Minor ATSv2 Charan Hebri Abhishek Modi
HDFS-13848 Refactor NameNode failover proxy providers Major ha, hdfs-client Konstantin Shvachko Konstantin Shvachko

OTHER:

JIRA Summary Priority Component Reporter Contributor
HDFS-13788 Update EC documentation about rack fault tolerance Major documentation, erasure-coding Xiao Chen Kitti Nanasi