Apache Hadoop Changelog

Release 2.9.2 - Unreleased (as of 2018-09-02)

NEW FEATURES:

JIRA Summary Priority Component Reporter Contributor
HDFS-13283 Percentage based Reserved Space Calculation for DataNode Major datanode, hdfs Lukas Majercak Lukas Majercak

IMPROVEMENTS:

JIRA Summary Priority Component Reporter Contributor
HADOOP-14987 Improve KMSClientProvider log around delegation token checking Major . Xiaoyu Yao Xiaoyu Yao
HADOOP-15394 Backport PowerShell NodeFencer HADOOP-14309 to branch-2 Minor . Íñigo Goiri Íñigo Goiri
HDFS-13462 Add BIND_HOST configuration for JournalNode’s HTTP and RPC Servers Major hdfs, journal-node Lukas Majercak Lukas Majercak
HADOOP-14841 Kms client should disconnect if unable to get output stream from connection. Major kms Xiao Chen Rushabh S Shah
HDFS-13272 DataNodeHttpServer to have configurable HttpServer2 threads Major datanode Erik Krogen Erik Krogen
HADOOP-15441 Log kms url and token service at debug level. Minor . Wei-Chiu Chuang Gabor Bota
HDFS-13544 Improve logging for JournalNode in federated cluster Major federation, hdfs Hanisha Koneru Hanisha Koneru
HADOOP-15486 Make NetworkTopology#netLock fair Major net Nanda kumar Nanda kumar
HADOOP-15449 Increase default timeout of ZK session to avoid frequent NameNode failover Critical common Karthik Palanisamy Karthik Palanisamy
HDFS-13602 Add checkOperation(WRITE) checks in FSNamesystem Major ha, namenode Erik Krogen Chao Sun
HDFS-13653 Make dfs.client.failover.random.order a per nameservice configuration Major federation Ekanth Sethuramalingam Ekanth Sethuramalingam
HDFS-13714 Fix TestNameNodePrunesMissingStorages test failures on Windows Major hdfs, namenode, test Lukas Majercak Lukas Majercak
HDFS-11060 make DEFAULT_MAX_CORRUPT_FILEBLOCKS_RETURNED configurable Minor hdfs Lantao Jin Lantao Jin
HDFS-13813 Exit NameNode if dangling child inode is detected when saving FsImage Major hdfs, namenode Siyao Meng Siyao Meng
HDFS-13821 RBF: Add dfs.federation.router.mount-table.cache.enable so that users can disable cache Major hdfs Fei Hui Fei Hui
HADOOP-15689 Add “*.patch” into .gitignore file of branch-2 Major . Rui Gao Rui Gao
HDFS-13854 RBF: The ProcessingAvgTime and ProxyAvgTime should display by JMX with ms unit. Major federation, hdfs yanghuafeng yanghuafeng
YARN-8051 TestRMEmbeddedElector#testCallbackSynchronization is flakey Major test Robert Kanter Robert Kanter

BUG FIXES:

JIRA Summary Priority Component Reporter Contributor
HADOOP-15121 Encounter NullPointerException when using DecayRpcScheduler Major . Tao Jie Tao Jie
HDFS-10803 TestBalancerWithMultipleNameNodes#testBalancing2OutOf3Blockpools fails intermittently due to no free space available Major . Yiqun Lin Yiqun Lin
YARN-8068 Application Priority field causes NPE in app timeline publish when Hadoop 2.7 based clients to 2.8+ Blocker yarn Sunil Govindan Sunil Govindan
HADOOP-15317 Improve NetworkTopology chooseRandom’s loop Major . Xiao Chen Xiao Chen
HADOOP-15375 Branch-2 pre-commit failed to build docker image Major . Xiao Chen Xiao Chen
HADOOP-15357 Configuration.getPropsWithPrefix no longer does variable substitution Major . Jim Brennan Jim Brennan
HDFS-7101 Potential null dereference in DFSck#doWork() Minor . Ted Yu skrho
YARN-8120 JVM can crash with SIGSEGV when exiting due to custom leveldb logger Major nodemanager, resourcemanager Jason Lowe Jason Lowe
YARN-8147 TestClientRMService#testGetApplications sporadically fails Major test Jason Lowe Jason Lowe
HADOOP-14970 MiniHadoopClusterManager doesn’t respect lack of format option Minor . Erik Krogen Erik Krogen
HDFS-12828 OIV ReverseXML Processor fails with escaped characters Critical hdfs Erik Krogen Erik Krogen
HADOOP-15180 branch-2 : daemon processes’ sysout overwrites ‘ulimit -a’ in daemon’s out file Minor scripts Ranith Sardar Ranith Sardar
HADOOP-15396 Some java source files are executable Minor . Akira Ajisaka Shashikant Banerjee
YARN-7786 NullPointerException while launching ApplicationMaster Major . lujie lujie
HDFS-10183 Prevent race condition during class initialization Minor fs Pavel Avgustinov Pavel Avgustinov
HDFS-13408 MiniDFSCluster to support being built on randomized base directory Major test Xiao Liang Xiao Liang
HADOOP-15390 Yarn RM logs flooded by DelegationTokenRenewer trying to renew KMS tokens Critical . Xiao Chen Xiao Chen
HDFS-13336 Test cases of TestWriteToReplica failed in windows Major . Xiao Liang Xiao Liang
HADOOP-15385 Many tests are failing in hadoop-distcp project in branch-2 Critical tools/distcp Rushabh S Shah Jason Lowe
HDFS-13509 Bug fix for breakHardlinks() of ReplicaInfo/LocalReplica, and fix TestFileAppend failures on Windows Major . Xiao Liang Xiao Liang
YARN-8232 RMContainer lost queue name when RM HA happens Major resourcemanager Hu Ziqian Hu Ziqian
HDFS-13537 TestHdfsHelper does not generate jceks path properly for relative path in Windows Major . Xiao Liang Xiao Liang
YARN-7003 DRAINING state of queues is not recovered after RM restart Major capacityscheduler Tao Yang Tao Yang
YARN-8244 TestContainerSchedulerQueuing.testStartMultipleContainers failed Major . Miklos Szegedi Jim Brennan
HDFS-13581 DN UI logs link is broken when https is enabled Minor datanode Namit Maheshwari Shashikant Banerjee
HDFS-13586 Fsync fails on directories on Windows Critical datanode, hdfs Lukas Majercak Lukas Majercak
HDFS-13590 Backport HDFS-12378 to branch-2 Major datanode, hdfs, test Lukas Majercak Lukas Majercak
HADOOP-15450 Avoid fsync storm triggered by DiskChecker and handle disk full situation Blocker . Kihwal Lee Arpit Agarwal
HDFS-13588 Fix TestFsDatasetImpl test failures on Windows Major . Xiao Liang Xiao Liang
YARN-8344 Missing nm.stop() in TestNodeManagerResync to fix testKillContainersOnResync Major . Giovanni Matteo Fumarola Giovanni Matteo Fumarola
YARN-8327 Fix TestAggregatedLogFormat#testReadAcontainerLogs1 on Windows Major log-aggregation Giovanni Matteo Fumarola Giovanni Matteo Fumarola
YARN-8346 Upgrading to 3.1 kills running containers with error “Opportunistic container queue is full” Blocker . Rohith Sharma K S Jason Lowe
HDFS-13618 Fix TestDataNodeFaultInjector test failures on Windows Major test Xiao Liang Xiao Liang
HADOOP-15473 Configure serialFilter in KeyProvider to avoid UnrecoverableKeyException caused by JDK-8189997 Critical kms Gabor Bota Gabor Bota
MAPREDUCE-7103 Fix TestHistoryViewerPrinter on windows due to a mismatch line separator Minor . Giovanni Matteo Fumarola Giovanni Matteo Fumarola
YARN-8359 Exclude containermanager.linux test classes on Windows Major . Giovanni Matteo Fumarola Jason Lowe
HDFS-13664 Refactor ConfiguredFailoverProxyProvider to make inheritance easier Minor hdfs-client Chao Sun Chao Sun
HDFS-13667 Typo: Marking all “datandoes” as stale Trivial namenode Wei-Chiu Chuang Nanda kumar
YARN-8405 RM zk-state-store.parent-path ACLs has been changed since HADOOP-14773 Major . Rohith Sharma K S Íñigo Goiri
MAPREDUCE-7108 TestFileOutputCommitter fails on Windows Minor test Zuoming Zhang Zuoming Zhang
HDFS-13675 Speed up TestDFSAdminWithHA Major hdfs, namenode Lukas Majercak Lukas Majercak
HDFS-13673 TestNameNodeMetrics fails on Windows Minor test Zuoming Zhang Zuoming Zhang
HDFS-13676 TestEditLogRace fails on Windows Minor test Zuoming Zhang Zuoming Zhang
HADOOP-15523 Shell command timeout given is in seconds whereas it is taken as millisec while scheduling Major . Bilwa S T Bilwa S T
YARN-8444 NodeResourceMonitor crashes on bad swapFree value Major . Jim Brennan Jim Brennan
YARN-8443 Total #VCores in cluster metrics is wrong when CapacityScheduler reserved some containers Major webapp Tao Yang Tao Yang
YARN-8457 Compilation is broken with -Pyarn-ui Major webapp Sunil Govindan Sunil Govindan
YARN-8401 [UI2] new ui is not accessible with out internet connection Blocker . Bibin A Chundatt Bibin A Chundatt
YARN-8451 Multiple NM heartbeat thread created when a slow NM resync with RM Major nodemanager Botong Huang Botong Huang
HADOOP-15548 Randomize local dirs Minor . Jim Brennan Jim Brennan
YARN-8473 Containers being launched as app tears down can leave containers in NEW state Major nodemanager Jason Lowe Jason Lowe
HDFS-13729 Fix broken links to RBF documentation Minor documentation jwhitter Gabor Bota
YARN-8518 test-container-executor test_is_empty() is broken Major . Jim Brennan Jim Brennan
YARN-8515 container-executor can crash with SIGPIPE after nodemanager restart Major . Jim Brennan Jim Brennan
YARN-8421 when moving app, activeUsers is increased, even though app does not have outstanding request Major . kyungwan nam
HADOOP-15614 TestGroupsCaching.testExceptionOnBackgroundRefreshHandled reliably fails Major . Kihwal Lee Weiwei Yang
YARN-8577 Fix the broken anchor in SLS site-doc Minor documentation Weiwei Yang Weiwei Yang
YARN-4606 CapacityScheduler: applications could get starved because computation of #activeUsers considers pending apps Critical capacity scheduler, capacityscheduler Karam Singh Manikandan R
HADOOP-15637 LocalFs#listLocatedStatus does not filter out hidden .crc files Minor fs Erik Krogen Erik Krogen
YARN-8331 Race condition in NM container launched after done Major . Yang Wang Pradeep Ambati
HDFS-13758 DatanodeManager should throw exception if it has BlockRecoveryCommand but the block is not under construction Major namenode Wei-Chiu Chuang chencan
YARN-8612 Fix NM Collector Service Port issue in YarnConfiguration Major ATSv2 Prabha Manepalli Prabha Manepalli
HADOOP-15674 Test failure TestSSLHttpServer.testExcludedCiphers with TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256 cipher suite Major common Gabor Bota Szilard Nemeth
YARN-8640 Restore previous state in container-executor after failure Major . Jim Brennan Jim Brennan
HADOOP-14314 The OpenSolaris taxonomy link is dead in InterfaceClassification.md Major documentation Daniel Templeton Rui Gao
YARN-8649 NPE in localizer hearbeat processing if a container is killed while localizing Major . lujie lujie

TESTS:

JIRA Summary Priority Component Reporter Contributor
HDFS-13337 Backport HDFS-4275 to branch-2.9 Minor . Íñigo Goiri Xiao Liang
HDFS-13503 Fix TestFsck test failures on Windows Major hdfs Xiao Liang Xiao Liang
HDFS-13542 TestBlockManager#testNeededReplicationWhileAppending fails due to improper cluster shutdown in TestBlockManager#testBlockManagerMachinesArray on Windows Minor . Anbang Hu Anbang Hu
HDFS-13551 TestMiniDFSCluster#testClusterSetStorageCapacity does not shut down cluster Minor . Anbang Hu Anbang Hu
HDFS-11700 TestHDFSServerPorts#testBackupNodePorts doesn’t pass on Windows Minor . Anbang Hu Anbang Hu
HDFS-13548 TestResolveHdfsSymlink#testFcResolveAfs fails on Windows Minor . Anbang Hu Anbang Hu
HDFS-13567 TestNameNodeMetrics#testGenerateEDEKTime,TestNameNodeMetrics#testResourceCheck should use a different cluster basedir Minor . Anbang Hu Anbang Hu
HDFS-13557 TestDFSAdmin#testListOpenFiles fails on Windows Minor . Anbang Hu Anbang Hu
HDFS-13550 TestDebugAdmin#testComputeMetaCommand fails on Windows Minor . Anbang Hu Anbang Hu
HDFS-13559 TestBlockScanner does not close TestContext properly Minor . Anbang Hu Anbang Hu
HDFS-13570 TestQuotaByStorageType,TestQuota,TestDFSOutputStream fail on Windows Minor . Anbang Hu Anbang Hu
HDFS-13558 TestDatanodeHttpXFrame does not shut down cluster Minor . Anbang Hu Anbang Hu
HDFS-13554 TestDatanodeRegistration#testForcedRegistration does not shut down cluster Minor . Anbang Hu Anbang Hu
HDFS-13556 TestNestedEncryptionZones does not shut down cluster Minor . Anbang Hu Anbang Hu
HDFS-13560 Insufficient system resources exist to complete the requested service for some tests on Windows Minor . Anbang Hu Anbang Hu
HDFS-13592 TestNameNodePrunesMissingStorages#testNameNodePrunesUnreportedStorages does not shut down cluster properly Minor . Anbang Hu Anbang Hu
HDFS-13593 TestBlockReaderLocalLegacy#testBlockReaderLocalLegacyWithAppend fails on Windows Minor test Anbang Hu Anbang Hu
HDFS-13587 TestQuorumJournalManager fails on Windows Minor . Anbang Hu Anbang Hu
HDFS-13620 Randomize the test directory path for TestHDFSFileSystemContract Minor . Anbang Hu Anbang Hu
HDFS-13591 TestDFSShell#testSetrepLow fails on Windows Minor . Anbang Hu Anbang Hu
HDFS-13632 Randomize baseDir for MiniJournalCluster in MiniQJMHACluster for TestDFSAdminWithHA Minor . Anbang Hu Anbang Hu
MAPREDUCE-7102 Fix TestJavaSerialization for Windows due a mismatch line separator Minor . Giovanni Matteo Fumarola Giovanni Matteo Fumarola
HDFS-13652 Randomize baseDir for MiniDFSCluster in TestBlockScanner Minor . Anbang Hu Anbang Hu
YARN-8370 Some Node Manager tests fail on Windows due to improper path/file separator Minor . Anbang Hu Anbang Hu
YARN-8422 TestAMSimulator failing with NPE Minor . Giovanni Matteo Fumarola Giovanni Matteo Fumarola
HADOOP-15532 TestBasicDiskValidator fails with NoSuchFileException Minor . Íñigo Goiri Giovanni Matteo Fumarola
HDFS-13563 TestDFSAdminWithHA times out on Windows Minor . Anbang Hu Lukas Majercak
HDFS-13681 Fix TestStartup.testNNFailToStartOnReadOnlyNNDir test failure on Windows Major test Xiao Liang Xiao Liang

SUB-TASKS:

JIRA Summary Priority Component Reporter Contributor
HDFS-13353 RBF: TestRouterWebHDFSContractCreate failed Major test Takanobu Asanuma Takanobu Asanuma
YARN-8110 AMRMProxy recover should catch for all throwable to avoid premature exit Major . Botong Huang Botong Huang
HDFS-13402 RBF: Fix java doc for StateStoreFileSystemImpl Minor hdfs Yiran Wu Yiran Wu
HDFS-13410 RBF: Support federation with no subclusters Minor . Íñigo Goiri Íñigo Goiri
HDFS-13384 RBF: Improve timeout RPC call mechanism Minor . Íñigo Goiri Íñigo Goiri
HDFS-13045 RBF: Improve error message returned from subcluster Minor . Wei Yan Íñigo Goiri
HDFS-13428 RBF: Remove LinkedList From StateStoreFileImpl.java Trivial federation BELUGA BEHR BELUGA BEHR
HDFS-13386 RBF: Wrong date information in list file(-ls) result Minor . Dibyendu Karmakar Dibyendu Karmakar
HADOOP-14999 AliyunOSS: provide one asynchronous multi-part based uploading mechanism Major fs/oss Genmao Yu Genmao Yu
HDFS-13435 RBF: Improve the error loggings for printing the stack trace Major . Yiqun Lin Yiqun Lin
YARN-7189 Container-executor doesn’t remove Docker containers that error out early Major yarn Eric Badger Eric Badger
HDFS-13466 RBF: Add more router-related information to the UI Minor . Wei Yan Wei Yan
HDFS-13453 RBF: getMountPointDates should fetch latest subdir time/date when parent dir is not present but /parent/child dirs are present in mount table Major . Dibyendu Karmakar Dibyendu Karmakar
HDFS-13478 RBF: Disabled Nameservice store API Major . Íñigo Goiri Íñigo Goiri
HDFS-13490 RBF: Fix setSafeMode in the Router Major . Íñigo Goiri Íñigo Goiri
HDFS-13484 RBF: Disable Nameservices from the federation Major . Íñigo Goiri Íñigo Goiri
HDFS-13326 RBF: Improve the interfaces to modify and view mount tables Minor . Wei Yan Gang Li
HDFS-13499 RBF: Show disabled name services in the UI Minor . Íñigo Goiri Íñigo Goiri
HDFS-13508 RBF: Normalize paths (automatically) when adding, updating, removing or listing mount table entries Minor . Ekanth Sethuramalingam Ekanth Sethuramalingam
HDFS-13434 RBF: Fix dead links in RBF document Major documentation Akira Ajisaka Chetna Chaudhari
HDFS-13488 RBF: Reject requests when a Router is overloaded Major . Íñigo Goiri Íñigo Goiri
HDFS-13525 RBF: Add unit test TestStateStoreDisabledNameservice Major . Yiqun Lin Yiqun Lin
YARN-8253 HTTPS Ats v2 api call fails with “bad HTTP parsed” Critical ATSv2 Yesha Vora Charan Hebri
HADOOP-15454 TestRollingFileSystemSinkWithLocal fails on Windows Major test Xiao Liang Xiao Liang
HADOOP-15498 TestHadoopArchiveLogs (#testGenerateScript, #testPrepareWorkingDir) fails on Windows Minor . Anbang Hu Anbang Hu
HDFS-13637 RBF: Router fails when threadIndex (in ConnectionPool) wraps around Integer.MIN_VALUE Critical federation CR Hota CR Hota
YARN-4781 Support intra-queue preemption for fairness ordering policy. Major scheduler Wangda Tan Eric Payne
HDFS-13281 Namenode#createFile should be /.reserved/raw/ aware. Critical encryption Rushabh S Shah Rushabh S Shah
YARN-4677 RMNodeResourceUpdateEvent update from scheduler can lead to race condition Major graceful, resourcemanager, scheduler Brook Zhou Wilfred Spiegelenburg
HADOOP-15529 ContainerLaunch#testInvalidEnvVariableSubstitutionType is not supported in Windows Minor . Giovanni Matteo Fumarola Giovanni Matteo Fumarola
HADOOP-15458 TestLocalFileSystem#testFSOutputStreamBuilder fails on Windows Minor test Xiao Liang Xiao Liang
HDFS-13475 RBF: Admin cannot enforce Router enter SafeMode Major . Wei Yan Chao Sun
HDFS-13733 RBF: Add Web UI configurations and descriptions to RBF document Minor documentation Takanobu Asanuma Takanobu Asanuma
HDFS-13743 RBF: Router throws NullPointerException due to the invalid initialization of MountTableResolver Major . Takanobu Asanuma Takanobu Asanuma
HDFS-13750 RBF: Router ID in RouterRpcClient is always null Major . Takanobu Asanuma Takanobu Asanuma
HDFS-13848 Refactor NameNode failover proxy providers Major ha, hdfs-client Konstantin Shvachko Konstantin Shvachko
HADOOP-15699 Fix some of testContainerManager failures in Windows Major . Botong Huang Botong Huang