Apache Hadoop Changelog

Release 2.7.2 - 2016-01-25

INCOMPATIBLE CHANGES:

JIRA Summary Priority Component Reporter Contributor
YARN-4087 Followup fixes after YARN-2019 regarding RM behavior when state-store error occurs Major . Jian He Jian He

IMPROVEMENTS:

JIRA Summary Priority Component Reporter Contributor
HDFS-7645 Rolling upgrade is restoring blocks from trash multiple times Major datanode Nathan Roberts Keisuke Ogiwara
YARN-3248 Display count of nodes blacklisted by apps in the web UI Major capacityscheduler, resourcemanager Varun Vasudev Varun Vasudev
HADOOP-11812 Implement listLocatedStatus for ViewFileSystem to speed up split calculation Blocker fs Gera Shegalov Gera Shegalov
HADOOP-7139 Allow appending to existing SequenceFiles Major io Stephen Rose Kanaka Kumar Avvaru
HDFS-8659 Block scanner INFO message is spamming logs Major datanode Yongjun Zhang Yongjun Zhang
HDFS-8722 Optimize datanode writes for small writes and flushes Critical . Kihwal Lee Kihwal Lee
HADOOP-12232 Upgrade Tomcat dependency to 6.0.44. Major build Chris Nauroth Chris Nauroth
YARN-3170 YARN architecture document needs updating Major documentation Allen Wittenauer Brahma Reddy Battula
HDFS-7314 When the DFSClient lease cannot be renewed, abort open-for-write files rather than the entire DFSClient Major . Ming Ma Ming Ma
YARN-3978 Configurably turn off the saving of container info in Generic AHS Major timelineserver, yarn Eric Payne Eric Payne
HADOOP-12280 Skip unit tests based on maven profile rather than NativeCodeLoader.isNativeCodeLoaded Minor test Masatake Iwasaki Masatake Iwasaki
HADOOP-5323 Trash documentation should describe its directory structure and configurations Minor documentation Suman Sehgal Weiwei Yang
HDFS-8384 Allow NN to startup if there are files having a lease but are not under construction Minor namenode Tsz Wo Nicholas Sze Jing Zhao
HADOOP-12413 AccessControlList should avoid calling getGroupNames in isUserInList with empty groups. Major security zhihai xu zhihai xu
YARN-4158 Remove duplicate close for LogWriter in AppLogAggregatorImpl#uploadLogsForContainers Minor nodemanager zhihai xu zhihai xu
YARN-3727 For better error recovery, check if the directory exists before using it for localization. Major nodemanager zhihai xu zhihai xu
HDFS-9221 HdfsServerConstants#ReplicaState#getState should avoid calling values() since it creates a temporary array Major performance Staffan Friberg Staffan Friberg
HDFS-9434 Recommission a datanode with 500k blocks may pause NN for 30 seconds Major namenode Tsz Wo Nicholas Sze Tsz Wo Nicholas Sze
HADOOP-12825 Log slow name resolutions Major . Sidharta Seethana Sidharta Seethana

BUG FIXES:

JIRA Summary Priority Component Reporter Contributor
HADOOP-9242 Duplicate surefire plugin config in hadoop-common Major test Andrey Klochkov Andrey Klochkov
HDFS-6945 BlockManager should remove a block from excessReplicateMap and decrement ExcessBlocks metric when the block is removed Critical namenode Akira Ajisaka Akira Ajisaka
HDFS-8046 Allow better control of getContentSummary Major . Kihwal Lee Kihwal Lee
YARN-2890 MiniYarnCluster should turn on timeline service if configured to do so Major . Mit Desai Mit Desai
HDFS-7725 Incorrect “nodes in service” metrics caused all writes to fail Major . Ming Ma Ming Ma
HDFS-8099 Change “DFSInputStream has been closed already” message to debug log level Minor hdfs-client Charles Lamb Charles Lamb
HADOOP-11491 HarFs incorrectly declared as requiring an authority Critical fs Gera Shegalov Brahma Reddy Battula
MAPREDUCE-5649 Reduce cannot use more than 2G memory for the final merge Major mrv2 stanley shi Gera Shegalov
HDFS-8219 setStoragePolicy with folder behavior is different after cluster restart Major . Peter Shi Surendra Singh Lilhore
MAPREDUCE-6273 HistoryFileManager should check whether summaryFile exists to avoid FileNotFoundException causing HistoryFileInfo into MOVE_FAILED state Minor jobhistoryserver zhihai xu zhihai xu
HDFS-8431 hdfs crypto class not found in Windows Critical scripts Sumana Sathish Anu Engineer
HDFS-7609 Avoid retry cache collision when Standby NameNode loading edits Critical namenode Carrey Zhan Ming Ma
MAPREDUCE-6377 JHS sorting on state column not working in webUi Minor jobhistoryserver Bibin A Chundatt zhihai xu
YARN-3780 Should use equals when compare Resource in RMNodeImpl#ReconnectNodeTransition Minor resourcemanager zhihai xu zhihai xu
HDFS-8615 Correct HTTP method in WebHDFS document Major documentation Akira Ajisaka Brahma Reddy Battula
YARN-3802 Two RMNodes for the same NodeId are used in RM sometimes after NM is reconnected. Major resourcemanager zhihai xu zhihai xu
MAPREDUCE-5948 org.apache.hadoop.mapred.LineRecordReader does not handle multibyte record delimiters well Critical . Kris Geusebroek Akira Ajisaka
HADOOP-8151 Error handling in snappy decompressor throws invalid exceptions Major io, native Todd Lipcon Matt Foley
HDFS-8656 Preserve compatibility of ClientProtocol#rollingUpgrade after finalization Critical rolling upgrades Andrew Wang Andrew Wang
YARN-3793 Several NPEs when deleting local files on NM recovery Major nodemanager Karthik Kambatla Varun Saxena
YARN-3508 Prevent processing preemption events on the main RM dispatcher Major resourcemanager, scheduler Jason Lowe Varun Saxena
MAPREDUCE-6425 ShuffleHandler passes wrong “base” parameter to getMapOutputInfo if mapId is not in the cache. Major mrv2, nodemanager zhihai xu zhihai xu
HADOOP-12186 ActiveStandbyElector shouldn’t call monitorLockNodeAsync multiple times Major ha zhihai xu zhihai xu
YARN-3690 [JDK8] ‘mvn site’ fails Major api, site Akira Ajisaka Brahma Reddy Battula
MAPREDUCE-6426 TestShuffleHandler#testGetMapOutputInfo is failing Major test Devaraj K zhihai xu
HADOOP-12191 Bzip2Factory is not thread safe Major io Jason Lowe Brahma Reddy Battula
HDFS-8767 RawLocalFileSystem.listStatus() returns null for UNIX pipefile Critical . Haohui Mai Kanaka Kumar Avvaru
YARN-3535 Scheduler must re-request container resources when RMContainer transitions from ALLOCATED to KILLED Critical capacityscheduler, fairscheduler, resourcemanager Peng Zhang Peng Zhang
YARN-3905 Application History Server UI NPEs when accessing apps run after RM restart Major timelineserver Eric Payne Eric Payne
YARN-3878 AsyncDispatcher can hang while stopping if it is configured for draining events on stop Critical . Varun Saxena Varun Saxena
YARN-2019 Retrospect on decision of making RM crashed if any exception throw in ZKRMStateStore Critical . Junping Du Jian He
HDFS-8806 Inconsistent metrics: number of missing blocks with replication factor 1 not properly cleared Major . Zhe Zhang Zhe Zhang
YARN-3967 Fetch the application report from the AHS if the RM does not know about it Major . Mit Desai Mit Desai
YARN-3925 ContainerLogsUtils#getContainerLogFile fails to read container log files from full disks. Critical nodemanager zhihai xu zhihai xu
YARN-3990 AsyncDispatcher may overloaded with RMAppNodeUpdateEvent when Node is connected/disconnected Critical resourcemanager Rohith Sharma K S Bibin A Chundatt
HDFS-8850 VolumeScanner thread exits with exception if there is no block pool to be scanned but there are suspicious blocks Major datanode Colin P. McCabe Colin P. McCabe
HADOOP-12304 Applications using FileContext fail with the default file system configured to be wasb/s3/etc. Blocker fs Chris Nauroth Chris Nauroth
HADOOP-11932 MetricsSinkAdapter hangs when being stopped Critical . Jian He Brahma Reddy Battula
YARN-3999 RM hangs on draining events Major . Jian He Jian He
HDFS-8879 Quota by storage type usage incorrectly initialized upon namenode restart Major namenode Kihwal Lee Xiaoyu Yao
YARN-4005 Completed container whose app is finished is not removed from NMStateStore Major . Jun Gong Jun Gong
YARN-4047 ClientRMService getApplications has high scheduler lock contention Major resourcemanager Jason Lowe Jason Lowe
HDFS-8891 HDFS concat should keep srcs order Blocker . Yong Zhang Yong Zhang
MAPREDUCE-6439 AM may fail instead of retrying if RM shuts down during the allocate call Critical . Anubhav Dhoot Anubhav Dhoot
HDFS-8852 HDFS architecture documentation of version 2.x is outdated about append write support Major documentation Hong Dai Thanh Ajith S
YARN-3857 Memory leak in ResourceManager with SIMPLE mode Critical resourcemanager mujunchao mujunchao
HDFS-8867 Enable optimized block reports Major . Rushabh S Shah Daryn Sharp
HDFS-8863 The remaining space check in BlockPlacementPolicyDefault is flawed Critical . Kihwal Lee Kihwal Lee
MAPREDUCE-6454 MapReduce doesn’t set the HADOOP_CLASSPATH for jar lib in distributed cache. Critical . Junping Du Junping Du
YARN-3896 RMNode transitioned from RUNNING to REBOOTED because its response id had not been reset synchronously Major resourcemanager Jun Gong Jun Gong
HDFS-8846 Add a unit test for INotify functionality across a layout version upgrade Major namenode Zhe Zhang Zhe Zhang
HDFS-8950 NameNode refresh doesn’t remove DataNodes that are no longer in the allowed list Major datanode, namenode Daniel Templeton Daniel Templeton
HADOOP-12359 hadoop fs -getmerge doc is wrong Major documentation Daniel Templeton Jagadesh Kiran N
HADOOP-10365 BufferedOutputStream in FileUtil#unpackEntries() should be closed in finally block Minor util Ted Yu Kiran Kumar M R
HDFS-8995 Flaw in registration bookeeping can make DN die on reconnect Critical . Kihwal Lee Kihwal Lee
HADOOP-12006 Remove unimplemented option for `hadoop fs -ls` from document in branch-2.7 Major documentation, fs Akira Ajisaka Akira Ajisaka
HADOOP-12213 Interrupted exception can occur when Client#stop is called Minor . Oleg Zhurakousky Kuhu Shukla
YARN-4103 RM WebServices missing scheme for appattempts logLinks Major . Jonathan Eagles Jonathan Eagles
YARN-4105 Capacity Scheduler headroom for DRF is wrong Major capacityscheduler Chang Li Chang Li
MAPREDUCE-6442 Stack trace is missing when error occurs in client protocol provider’s constructor Major client Chang Li Chang Li
YARN-4096 App local logs are leaked if log aggregation fails to initialize for the app Major log-aggregation, nodemanager Jason Lowe Jason Lowe
HDFS-9033 dfsadmin -metasave prints “NaN” for cache used% Major . Archana T Brahma Reddy Battula
MAPREDUCE-6474 ShuffleHandler can possibly exhaust nodemanager file descriptors Major mrv2, nodemanager Nathan Roberts Kuhu Shukla
HDFS-9042 Update document for the Storage policy name Minor documentation J.Andreina J.Andreina
MAPREDUCE-6472 MapReduce AM should have java.io.tmpdir=./tmp to be consistent with tasks Major mr-am Jason Lowe Naganarasimha G R
MAPREDUCE-6481 LineRecordReader may give incomplete record and wrong position/key information for uncompressed input sometimes. Critical mrv2 zhihai xu zhihai xu
MAPREDUCE-5982 Task attempts that fail from the ASSIGNED state can disappear Major mr-am Jason Lowe Chang Li
YARN-3697 FairScheduler: ContinuousSchedulingThread can fail to shutdown Critical fairscheduler zhihai xu zhihai xu
HDFS-9043 Doc updation for commands in HDFS Federation Minor documentation J.Andreina J.Andreina
YARN-3975 WebAppProxyServlet should not redirect to RM page if AHS is enabled Major . Mit Desai Mit Desai
YARN-3624 ApplicationHistoryServer reverses the order of the filters it gets Major timelineserver Mit Desai Mit Desai
HDFS-9106 Transfer failure during pipeline recovery causes permanent write failures Critical . Kihwal Lee Kihwal Lee
MAPREDUCE-6492 AsyncDispatcher exit with NPE on TaskAttemptImpl#sendJHStartEventForAssignedFailTask Critical . Bibin A Chundatt Bibin A Chundatt
YARN-4180 AMLauncher does not retry on failures when talking to NM Critical resourcemanager Anubhav Dhoot Anubhav Dhoot
MAPREDUCE-6497 Fix wrong value of JOB_FINISHED event in JobHistoryEventHandler Major . Shinichi Yamashita Shinichi Yamashita
HADOOP-12230 hadoop-project declares duplicate, conflicting curator dependencies Minor build Steve Loughran Rakesh R
YARN-3619 ContainerMetrics unregisters during getMetrics and leads to ConcurrentModificationException Major nodemanager Jason Lowe zhihai xu
YARN-4209 RMStateStore FENCED state doesn’t work due to updateFencedState called by stateMachine.doTransition Critical resourcemanager zhihai xu zhihai xu
HDFS-9178 Slow datanode I/O can cause a wrong node to be marked bad Critical . Kihwal Lee Kihwal Lee
HADOOP-12465 Incorrect javadoc in WritableUtils.java Minor documentation Martin Petricek Jagadesh Kiran N
HDFS-8676 Delayed rolling upgrade finalization can cause heartbeat expiration and write failures Critical . Kihwal Lee Walter Su
HDFS-9220 Reading small file (< 512 bytes) that is open for append fails due to incorrect checksum Blocker . Bogdan Raducanu Jing Zhao
HADOOP-12464 Interrupted client may try to fail-over and retry Major ipc Kihwal Lee Kihwal Lee
YARN-4281 2.7 RM app page is broken Blocker . Chang Li Chang Li
YARN-3798 ZKRMStateStore shouldn’t create new session without occurrance of SESSIONEXPIED Blocker resourcemanager Bibin A Chundatt Varun Saxena
MAPREDUCE-6518 Set SO_KEEPALIVE on shuffle connections Major mrv2, nodemanager Nathan Roberts Chang Li
HDFS-9273 ACLs on root directory may be lost after NN restart Critical namenode Xiao Chen Xiao Chen
YARN-4000 RM crashes with NPE if leaf queue becomes parent queue during restart Major capacityscheduler, resourcemanager Jason Lowe Varun Saxena
YARN-4009 CORS support for ResourceManager REST API Major . Prakash Ramachandran Varun Vasudev
YARN-4041 Slow delegation token renewal can severely prolong RM recovery Major resourcemanager Jason Lowe Sunil G
HDFS-9290 DFSClient#callAppend() is not backward compatible for slightly older NameNodes Blocker . Tony Wu Tony Wu
HDFS-9305 Delayed heartbeat processing causes storm of subsequent heartbeats Major datanode Chris Nauroth Arpit Agarwal
HDFS-9317 Document fsck -blockId and -storagepolicy options in branch-2.7 Major documentation Akira Ajisaka Akira Ajisaka
HDFS-9083 Replication violates block placement policy. Blocker namenode Rushabh S Shah Rushabh S Shah
YARN-4313 Race condition in MiniMRYarnCluster when getting history server address Major . Jian He Jian He
YARN-4312 TestSubmitApplicationWithRMHA fails on branch-2.7 and branch-2.6 as some of the test cases time out Major . Varun Saxena Varun Saxena
YARN-4320 TestJobHistoryEventHandler fails as AHS in MiniYarnCluster no longer binds to default port 8188 Major . Varun Saxena Varun Saxena
MAPREDUCE-6528 Memory leak for HistoryFileManager.getJobSummary() Critical jobhistoryserver Junping Du Junping Du
MAPREDUCE-6451 DistCp has incorrect chunkFilePath for multiple jobs when strategy is dynamic Major distcp Kuhu Shukla Kuhu Shukla
YARN-4321 Incessant retries if NoAuthException is thrown by Zookeeper in non HA mode Major resourcemanager Varun Saxena Varun Saxena
YARN-4326 Fix TestDistributedShell timeout as AHS in MiniYarnCluster no longer binds to default port 8188 Major . MENG DING MENG DING
HDFS-9289 Make DataStreamer#block thread safe and verify genStamp in commitBlock Critical . Chang Li Chang Li
YARN-4127 RM fail with noAuth error if switched from failover mode to non-failover mode Major resourcemanager Jian He Varun Saxena
HADOOP-12526 [Branch-2] there are duplicate dependency definitions in pom’s Major build Sangjin Lee Sangjin Lee
HADOOP-12451 [Branch-2] Setting HADOOP_HOME explicitly should be allowed Blocker scripts Karthik Kambatla Karthik Kambatla
YARN-4241 Fix typo of property name in yarn-default.xml Major documentation Anthony Rojas Anthony Rojas
MAPREDUCE-6540 TestMRTimelineEventHandling fails Major test Sangjin Lee Sangjin Lee
YARN-4354 Public resource localization fails with NPE Blocker nodemanager Jason Lowe Jason Lowe
HDFS-9413 getContentSummary() on standby should throw StandbyException Critical . Brahma Reddy Battula Brahma Reddy Battula
HDFS-9431 DistributedFileSystem#concat fails if the target path is relative. Major hdfs-client Kazuho Fujii Kazuho Fujii
YARN-2859 ApplicationHistoryServer binds to default port 8188 in MiniYARNCluster Critical timelineserver Hitesh Shah Vinod Kumar Vavilapalli
HADOOP-12577 Bump up commons-collections version to 3.2.2 to address a security flaw Blocker build, security Wei-Chiu Chuang Wei-Chiu Chuang
YARN-4344 NMs reconnecting with changed capabilities can lead to wrong cluster resource calculations Critical resourcemanager Varun Vasudev Varun Vasudev
MAPREDUCE-5883 “Total megabyte-seconds” in job counters is slightly misleading Minor . Nathan Roberts Nathan Roberts
YARN-4365 FileSystemNodeLabelStore should check for root dir existence on startup Major resourcemanager Jason Lowe Kuhu Shukla
HADOOP-12415 hdfs and nfs builds broken on -missing compile-time dependency on netty Major nfs Konstantin Boudnik Tom Zeng
MAPREDUCE-6549 multibyte delimiters with LineRecordReader cause duplicate records Major mrv1, mrv2 Dustin Cote Wilfred Spiegelenburg
HDFS-9426 Rollingupgrade finalization is not backward compatible Blocker . Kihwal Lee Kihwal Lee
HDFS-9470 Encryption zone on root not loaded from fsimage after NN restart Critical . Xiao Chen Xiao Chen
HDFS-9294 DFSClient deadlock when close file and failed to renew lease Blocker hdfs-client DENG FEI Brahma Reddy Battula
YARN-4348 ZKRMStateStore.syncInternal shouldn’t wait for sync completion for avoiding blocking ZK’s event thread Blocker . Tsuyoshi Ozawa Tsuyoshi Ozawa
YARN-4424 Fix deadlock in RMAppImpl Blocker . Yesha Vora Jian He
YARN-4434 NodeManager Disk Checker parameter documentation is not correct Minor documentation, nodemanager Takashi Ohnishi Weiwei Yang
HDFS-9445 Datanode may deadlock while handling a bad volume Blocker . Kihwal Lee Walter Su
HDFS-9574 Reduce client failures during datanode restart Major . Kihwal Lee Kihwal Lee

TESTS:

JIRA Summary Priority Component Reporter Contributor
YARN-3580 [JDK 8] TestClientRMService.testGetLabelsToNodes fails Major test Robert Kanter Robert Kanter

SUB-TASKS:

JIRA Summary Priority Component Reporter Contributor
YARN-3136 getTransferredContainers can be a bottleneck during AM registration Major scheduler Jason Lowe Sunil G
YARN-3700 ATS Web Performance issue at load time when large number of jobs Major resourcemanager, webapp, yarn Xuan Gong Xuan Gong
YARN-3740 Fixed the typo with the configuration name: APPLICATION_HISTORY_PREFIX_MAX_APPS Major resourcemanager, webapp, yarn Xuan Gong Xuan Gong
YARN-3969 Allow jobs to be submitted to reservation that is active but does not have any allocations Major capacityscheduler, fairscheduler, resourcemanager Subru Krishnan Subru Krishnan
YARN-4092 RM HA UI redirection needs to be fixed when both RMs are in standby mode Major resourcemanager Xuan Gong Xuan Gong
YARN-2801 Add documentation for node labels feature Major documentation Gururaj Shetty Wangda Tan
YARN-3893 Both RM in active state when Admin#transitionToActive failure from refeshAll() Critical resourcemanager Bibin A Chundatt Bibin A Chundatt
YARN-4101 RM should print alert messages if Zookeeper and Resourcemanager gets connection issue Critical yarn Yesha Vora Xuan Gong
YARN-2513 Host framework UIs in YARN for use with the ATS Major timelineserver Jonathan Eagles Jonathan Eagles
YARN-2902 Killing a container that is localizing can orphan resources in the DOWNLOADING state Major nodemanager Jason Lowe Varun Saxena