“Apache Hadoop” Changelog

Release 2.8.4 - 2018-05-08

INCOMPATIBLE CHANGES:

JIRA Summary Priority Component Reporter Contributor

IMPORTANT ISSUES:

JIRA Summary Priority Component Reporter Contributor

NEW FEATURES:

JIRA Summary Priority Component Reporter Contributor

IMPROVEMENTS:

JIRA Summary Priority Component Reporter Contributor
HADOOP-12817 Enable TLS v1.1 and 1.2 Major security Robert Kanter Robert Kanter
HDFS-11409 DatanodeInfo getNetworkLocation and setNetworkLocation shoud use volatile instead of synchronized Minor performance Chen Liang Chen Liang
HADOOP-14246 Authentication Tokens should use SecureRandom instead of Random and 256 bit secrets Major security Robert Kanter Robert Kanter
HADOOP-14920 KMSClientProvider won’t work with KMS delegation token retrieved from non-Java client. Major kms Xiaoyu Yao Xiaoyu Yao
HADOOP-14987 Improve KMSClientProvider log around delegation token checking Major . Xiaoyu Yao Xiaoyu Yao
YARN-7495 Improve robustness of the AggregatedLogDeletionService Major log-aggregation Jonathan Eagles Jonathan Eagles
YARN-7678 Ability to enable logging of container memory stats Major nodemanager Jim Brennan Jim Brennan
YARN-7590 Improve container-executor validation check Major security, yarn Eric Yang Eric Yang
HADOOP-15212 Add independent secret manager method for logging expired tokens Major security Daryn Sharp Daryn Sharp
YARN-7728 Expose container preemptions related information in Capacity Scheduler queue metrics Major . Eric Payne Eric Payne
MAPREDUCE-7048 Uber AM can crash due to unknown task in statusUpdate Major mr-am Peter Bacsko Peter Bacsko
HADOOP-13972 ADLS to support per-store configuration Major fs/adl John Zhuge Sharad Sonker
YARN-7813 Capacity Scheduler Intra-queue Preemption should be configurable for each queue Major capacity scheduler, scheduler preemption Eric Payne Eric Payne
HDFS-11187 Optimize disk access for last partial chunk checksum of Finalized replica Major datanode Wei-Chiu Chuang Gabor Bota
HADOOP-15279 increase maven heap size recommendations Minor build, documentation, test Allen Wittenauer Allen Wittenauer
HDFS-12884 BlockUnderConstructionFeature.truncateBlock should be of type BlockInfo Major namenode Konstantin Shvachko chencan
HADOOP-14841 Kms client should disconnect if unable to get output stream from connection. Major kms Xiao Chen Rushabh S Shah

BUG FIXES:

JIRA Summary Priority Component Reporter Contributor
HADOOP-13988 KMSClientProvider does not work with WebHDFS and Apache Knox w/ProxyUser Major common, kms Greg Senia Xiaoyu Yao
HADOOP-14029 Fix KMSClientProvider for non-secure proxyuser use case Major kms Xiaoyu Yao Xiaoyu Yao
HDFS-12614 FSPermissionChecker#getINodeAttrs() throws NPE when INodeAttributesProvider configured Major . Manoj Govindassamy Manoj Govindassamy
YARN-7489 ConcurrentModificationException in RMAppImpl#getRMAppMetrics Major capacityscheduler Tao Yang Tao Yang
YARN-7525 Incorrect query parameters in cluster nodes REST API document Minor documentation Tao Yang Tao Yang
HDFS-12754 Lease renewal can hit a deadlock Major . Kuhu Shukla Kuhu Shukla
HDFS-11754 Make FsServerDefaults cache configurable. Minor . Rushabh S Shah Mikhail Erofeev
MAPREDUCE-5124 AM lacks flow control for task events Major mr-am Jason Lowe Peter Bacsko
HDFS-12833 Distcp : Update the usage of delete option for dependency with update and overwrite option Minor distcp, hdfs Harshakiran Reddy usharani
HDFS-12907 Allow read-only access to reserved raw for non-superusers Major namenode Daryn Sharp Rushabh S Shah
HDFS-12881 Output streams closed with IOUtils suppressing write errors Major . Jason Lowe Ajay Kumar
YARN-7595 Container launching code suppresses close exceptions after writes Major nodemanager Jason Lowe Jim Brennan
YARN-7661 NodeManager metrics return wrong value after update node resource Major . Yang Wang Yang Wang
HDFS-12347 TestBalancerRPCDelay#testBalancerRPCDelay fails very frequently Critical test Xiao Chen Bharat Viswanadham
HADOOP-15143 NPE due to Invalid KerberosTicket in UGI Major . Jitendra Nath Pandey Mukul Kumar Singh
MAPREDUCE-7028 Concurrent task progress updates causing NPE in Application Master Blocker mr-am Gergo Repas Gergo Repas
YARN-7619 Max AM Resource value in Capacity Scheduler UI has to be refreshed for every user Major capacity scheduler, yarn Eric Payne Eric Payne
YARN-7663 RMAppImpl:Invalid event: START at KILLED Minor resourcemanager lujie lujie
YARN-6948 Invalid event: ATTEMPT_ADDED at FINAL_SAVING Minor yarn lujie lujie
HDFS-6804 Add test for race condition between transferring block and appending block causes “Unexpected checksum mismatch exception” Major datanode Gordon Wang Brahma Reddy Battula
HDFS-9049 Make Datanode Netty reverse proxy port to be configurable Major datanode Vinayakumar B Vinayakumar B
HADOOP-15150 in FsShell, UGI params should be overidden through env vars(-D arg) Major . Brahma Reddy Battula Brahma Reddy Battula
HADOOP-15181 Typo in SecureMode.md Trivial documentation Masahiro Tanaka Masahiro Tanaka
YARN-7102 NM heartbeat stuck when responseId overflows MAX_INT Critical . Botong Huang Botong Huang
MAPREDUCE-7020 Task timeout in uber mode can crash AM Major mr-am Akira Ajisaka Peter Bacsko
HDFS-13100 Handle IllegalArgumentException when GETSERVERDEFAULTS is not implemented in webhdfs. Critical hdfs, webhdfs Yongjun Zhang Yongjun Zhang
YARN-7849 TestMiniYarnClusterNodeUtilization#testUpdateNodeUtilization fails due to heartbeat sync error Major test Jason Lowe Botong Huang
HDFS-13120 Snapshot diff could be corrupted after concat Major namenode, snapshots Xiaoyu Yao Xiaoyu Yao
HDFS-10453 ReplicationMonitor thread could stuck for long time due to the race between replication and delete of same file in a large cluster. Major namenode He Xiaoqiao He Xiaoqiao
HDFS-8693 refreshNamenodes does not support adding a new standby to a running DN Critical datanode, ha Jian Fang Ajith S
MAPREDUCE-7052 TestFixedLengthInputFormat#testFormatCompressedIn is flaky Major client, test Peter Bacsko Peter Bacsko
HDFS-13112 Token expiration edits may cause log corruption or deadlock Critical namenode Daryn Sharp Daryn Sharp
MAPREDUCE-7053 Timed out tasks can fail to produce thread dump Major . Jason Lowe Jason Lowe
HADOOP-15206 BZip2 drops and duplicates records when input split size is small Major . Aki Tanaka Aki Tanaka
YARN-7947 Capacity Scheduler intra-queue preemption can NPE for non-schedulable apps Major capacity scheduler, scheduler preemption Eric Payne Eric Payne
HADOOP-14903 Add json-smart explicitly to pom.xml Major common Ray Chiang Ray Chiang
HDFS-12070 Failed block recovery leaves files open indefinitely and at risk for data loss Major . Daryn Sharp Kihwal Lee
HADOOP-15283 Upgrade from findbugs 3.0.1 to spotbugs 3.1.2 in branch-2 to fix docker image build Major . Xiao Chen Akira Ajisaka
HDFS-13164 File not closed if streamer fail with DSQuotaExceededException Major hdfs-client Xiao Chen Xiao Chen
HDFS-13109 Support fully qualified hdfs path in EZ commands Major hdfs Hanisha Koneru Hanisha Koneru
MAPREDUCE-6930 mapreduce.map.cpu.vcores and mapreduce.reduce.cpu.vcores are both present twice in mapred-default.xml Major mrv2 Daniel Templeton Sen Zhao
HDFS-12156 TestFSImage fails without -Pnative Major test Akira Ajisaka Akira Ajisaka
YARN-8054 Improve robustness of the LocalDirsHandlerService MonitoringTimerTask thread Major . Jonathan Eagles Jonathan Eagles
HDFS-13195 DataNode conf page cannot display the current value after reconfig Minor datanode maobaolong maobaolong
HADOOP-12862 LDAP Group Mapping over SSL can not specify trust store Major . Wei-Chiu Chuang Wei-Chiu Chuang
HADOOP-15317 Improve NetworkTopology chooseRandom’s loop Major . Xiao Chen Xiao Chen
HDFS-13427 Fix the section titles of transparent encryption document Minor documentation Akira Ajisaka Akira Ajisaka
YARN-8120 JVM can crash with SIGSEGV when exiting due to custom leveldb logger Major nodemanager, resourcemanager Jason Lowe Jason Lowe
YARN-8147 TestClientRMService#testGetApplications sporadically fails Major test Jason Lowe Jason Lowe
HADOOP-14970 MiniHadoopClusterManager doesn’t respect lack of format option Minor . Erik Krogen Erik Krogen
HADOOP-15180 branch-2 : daemon processes’ sysout overwrites ‘ulimit -a’ in daemon’s out file Minor scripts Ranith Sardar Ranith Sardar
HDFS-10183 Prevent race condition during class initialization Minor fs Pavel Avgustinov Pavel Avgustinov
HADOOP-15390 Yarn RM logs flooded by DelegationTokenRenewer trying to renew KMS tokens Critical . Xiao Chen Xiao Chen
HADOOP-15385 Many tests are failing in hadoop-distcp project in branch-2 Critical tools/distcp Rushabh S Shah Jason Lowe
MAPREDUCE-7073 Optimize TokenCache#obtainTokensForNamenodesInternal Major . Bibin A Chundatt Bibin A Chundatt

TESTS:

JIRA Summary Priority Component Reporter Contributor

SUB-TASKS:

JIRA Summary Priority Component Reporter Contributor
HADOOP-13174 Add more debug logs for delegation tokens and authentication Minor security Xiao Chen Xiao Chen
HADOOP-14799 Update nimbus-jose-jwt to 4.41.1 Major . Ray Chiang Ray Chiang
HDFS-12396 Webhdfs file system should get delegation token from kms provider. Major encryption, kms, webhdfs Rushabh S Shah Rushabh S Shah
HDFS-12574 Add CryptoInputStream to WebHdfsFileSystem read call. Major encryption, kms, webhdfs Rushabh S Shah Rushabh S Shah

OTHER:

JIRA Summary Priority Component Reporter Contributor
HADOOP-15177 Update the release year to 2018 Blocker build Akira Ajisaka Bharat Viswanadham