Apache Hadoop Changelog

Release 2.7.2 - 2016-01-25


JIRA Summary Priority Component Reporter Contributor
HDFS-7645 Rolling upgrade is restoring blocks from trash multiple times Major datanode Nathan Roberts Keisuke Ogiwara


JIRA Summary Priority Component Reporter Contributor


JIRA Summary Priority Component Reporter Contributor


JIRA Summary Priority Component Reporter Contributor
HADOOP-12825 Log slow name resolutions Major . Sidharta Seethana Sidharta Seethana
HADOOP-12413 AccessControlList should avoid calling getGroupNames in isUserInList with empty groups. Major security zhihai xu zhihai xu
HADOOP-12280 Skip unit tests based on maven profile rather than NativeCodeLoader.isNativeCodeLoaded Minor test Masatake Iwasaki Masatake Iwasaki
HADOOP-12232 Upgrade Tomcat dependency to 6.0.44. Major build Chris Nauroth Chris Nauroth
HADOOP-11812 Implement listLocatedStatus for ViewFileSystem to speed up split calculation Blocker fs Gera Shegalov Gera Shegalov
HADOOP-7139 Allow appending to existing SequenceFiles Major io Stephen Rose Kanaka Kumar Avvaru
HADOOP-5323 Trash documentation should describe its directory structure and configurations Minor documentation Suman Sehgal Weiwei Yang
HDFS-9434 Recommission a datanode with 500k blocks may pause NN for 30 seconds Major namenode Tsz Wo Nicholas Sze Tsz Wo Nicholas Sze
HDFS-9221 HdfsServerConstants#ReplicaState#getState should avoid calling values() since it creates a temporary array Major performance Staffan Friberg Staffan Friberg
HDFS-8722 Optimize datanode writes for small writes and flushes Critical . Kihwal Lee Kihwal Lee
HDFS-8659 Block scanner INFO message is spamming logs Major datanode Yongjun Zhang Yongjun Zhang
HDFS-8384 Allow NN to startup if there are files having a lease but are not under construction Minor namenode Tsz Wo Nicholas Sze Jing Zhao
HDFS-7314 When the DFSClient lease cannot be renewed, abort open-for-write files rather than the entire DFSClient Major . Ming Ma Ming Ma
YARN-4158 Remove duplicate close for LogWriter in AppLogAggregatorImpl#uploadLogsForContainers Minor nodemanager zhihai xu zhihai xu
YARN-3978 Configurably turn off the saving of container info in Generic AHS Major timelineserver, yarn Eric Payne Eric Payne
YARN-3727 For better error recovery, check if the directory exists before using it for localization. Major nodemanager zhihai xu zhihai xu
YARN-3248 Display count of nodes blacklisted by apps in the web UI Major capacityscheduler, resourcemanager Varun Vasudev Varun Vasudev
YARN-3170 YARN architecture document needs updating Major documentation Allen Wittenauer Brahma Reddy Battula


JIRA Summary Priority Component Reporter Contributor
HADOOP-12577 Bump up commons-collections version to 3.2.2 to address a security flaw Blocker build, security Wei-Chiu Chuang Wei-Chiu Chuang
HADOOP-12526 [Branch-2] there are duplicate dependency definitions in pom’s Major build Sangjin Lee Sangjin Lee
HADOOP-12465 Incorrect javadoc in WritableUtils.java Minor documentation Martin Petricek Jagadesh Kiran N
HADOOP-12464 Interrupted client may try to fail-over and retry Major ipc Kihwal Lee Kihwal Lee
HADOOP-12451 [Branch-2] Setting HADOOP_HOME explicitly should be allowed Blocker scripts Karthik Kambatla Karthik Kambatla
HADOOP-12415 hdfs and nfs builds broken on -missing compile-time dependency on netty Major nfs Konstantin Boudnik Tom Zeng
HADOOP-12359 hadoop fs -getmerge doc is wrong Major documentation Daniel Templeton Jagadesh Kiran N
HADOOP-12304 Applications using FileContext fail with the default file system configured to be wasb/s3/etc. Blocker fs Chris Nauroth Chris Nauroth
HADOOP-12230 hadoop-project declares duplicate, conflicting curator dependencies Minor build Steve Loughran Rakesh R
HADOOP-12213 Interrupted exception can occur when Client#stop is called Minor . Oleg Zhurakousky Kuhu Shukla
HADOOP-12191 Bzip2Factory is not thread safe Major io Jason Lowe Brahma Reddy Battula
HADOOP-12186 ActiveStandbyElector shouldn’t call monitorLockNodeAsync multiple times Major ha zhihai xu zhihai xu
HADOOP-12006 Remove unimplemented option for hadoop fs -ls from document in branch-2.7 Major . Akira AJISAKA Akira AJISAKA
HADOOP-11932 MetricsSinkAdapter hangs when being stopped Critical . Jian He Brahma Reddy Battula
HADOOP-11491 HarFs incorrectly declared as requiring an authority Critical fs Gera Shegalov Brahma Reddy Battula
HADOOP-10365 BufferedOutputStream in FileUtil#unpackEntries() should be closed in finally block Minor util Ted Yu Kiran Kumar M R
HADOOP-9242 Duplicate surefire plugin config in hadoop-common Major test Andrey Klochkov Andrey Klochkov
HADOOP-8151 Error handling in snappy decompressor throws invalid exceptions Major io, native Todd Lipcon Matt Foley
HDFS-9574 Reduce client failures during datanode restart Major . Kihwal Lee Kihwal Lee
HDFS-9470 Encryption zone on root not loaded from fsimage after NN restart Critical . Xiao Chen Xiao Chen
HDFS-9445 Datanode may deadlock while handling a bad volume Blocker . Kihwal Lee Walter Su
HDFS-9431 DistributedFileSystem#concat fails if the target path is relative. Major hdfs-client Kazuho Fujii Kazuho Fujii
HDFS-9426 Rollingupgrade finalization is not backward compatible Blocker . Kihwal Lee Kihwal Lee
HDFS-9413 getContentSummary() on standby should throw StandbyException Critical . Brahma Reddy Battula Brahma Reddy Battula
HDFS-9317 Document fsck -blockId and -storagepolicy options in branch-2.7 Major documentation Akira AJISAKA Akira AJISAKA
HDFS-9305 Delayed heartbeat processing causes storm of subsequent heartbeats Major datanode Chris Nauroth Arpit Agarwal
HDFS-9294 DFSClient deadlock when close file and failed to renew lease Blocker hdfs-client DENG FEI Brahma Reddy Battula
HDFS-9290 DFSClient#callAppend() is not backward compatible for slightly older NameNodes Blocker . Tony Wu Tony Wu
HDFS-9289 Make DataStreamer#block thread safe and verify genStamp in commitBlock Critical . Chang Li Chang Li
HDFS-9273 ACLs on root directory may be lost after NN restart Critical namenode Xiao Chen Xiao Chen
HDFS-9220 Reading small file (< 512 bytes) that is open for append fails due to incorrect checksum Blocker . Bogdan Raducanu Jing Zhao
HDFS-9178 Slow datanode I/O can cause a wrong node to be marked bad Critical . Kihwal Lee Kihwal Lee
HDFS-9106 Transfer failure during pipeline recovery causes permanent write failures Critical . Kihwal Lee Kihwal Lee
HDFS-9083 Replication violates block placement policy. Blocker namenode Rushabh S Shah Rushabh S Shah
HDFS-9043 Doc updation for commands in HDFS Federation Minor documentation J.Andreina J.Andreina
HDFS-9042 Update document for the Storage policy name Minor documentation J.Andreina J.Andreina
HDFS-9033 dfsadmin -metasave prints “NaN” for cache used% Major . Archana T Brahma Reddy Battula
HDFS-8995 Flaw in registration bookeeping can make DN die on reconnect Critical . Kihwal Lee Kihwal Lee
HDFS-8950 NameNode refresh doesn’t remove DataNodes that are no longer in the allowed list Major datanode, namenode Daniel Templeton Daniel Templeton
HDFS-8891 HDFS concat should keep srcs order Blocker . Yong Zhang Yong Zhang
HDFS-8879 Quota by storage type usage incorrectly initialized upon namenode restart Major namenode Kihwal Lee Xiaoyu Yao
HDFS-8867 Enable optimized block reports Major . Rushabh S Shah Daryn Sharp
HDFS-8863 The remaining space check in BlockPlacementPolicyDefault is flawed Critical . Kihwal Lee Kihwal Lee
HDFS-8852 HDFS architecture documentation of version 2.x is outdated about append write support Major documentation Hong Dai Thanh Ajith S
HDFS-8850 VolumeScanner thread exits with exception if there is no block pool to be scanned but there are suspicious blocks Major datanode Colin Patrick McCabe Colin Patrick McCabe
HDFS-8846 Add a unit test for INotify functionality across a layout version upgrade Major namenode Zhe Zhang Zhe Zhang
HDFS-8806 Inconsistent metrics: number of missing blocks with replication factor 1 not properly cleared Major . Zhe Zhang Zhe Zhang
HDFS-8767 RawLocalFileSystem.listStatus() returns null for UNIX pipefile Critical . Haohui Mai Kanaka Kumar Avvaru
HDFS-8676 Delayed rolling upgrade finalization can cause heartbeat expiration and write failures Critical . Kihwal Lee Walter Su
HDFS-8656 Preserve compatibility of ClientProtocol#rollingUpgrade after finalization Critical rolling upgrades Andrew Wang Andrew Wang
HDFS-8615 Correct HTTP method in WebHDFS document Major documentation Akira AJISAKA Brahma Reddy Battula
HDFS-8431 hdfs crypto class not found in Windows Critical scripts Sumana Sathish Anu Engineer
HDFS-8219 setStoragePolicy with folder behavior is different after cluster restart Major . Peter Shi Surendra Singh Lilhore
HDFS-8099 Change “DFSInputStream has been closed already” message to debug log level Minor hdfs-client Charles Lamb Charles Lamb
HDFS-8046 Allow better control of getContentSummary Major . Kihwal Lee Kihwal Lee
HDFS-7725 Incorrect “nodes in service” metrics caused all writes to fail Major . Ming Ma Ming Ma
HDFS-7609 Avoid retry cache collision when Standby NameNode loading edits Critical namenode Carrey Zhan Ming Ma
HDFS-6945 BlockManager should remove a block from excessReplicateMap and decrement ExcessBlocks metric when the block is removed Critical namenode Akira AJISAKA Akira AJISAKA
MAPREDUCE-6549 multibyte delimiters with LineRecordReader cause duplicate records Major mrv1, mrv2 Dustin Cote Wilfred Spiegelenburg
MAPREDUCE-6540 TestMRTimelineEventHandling fails Major test Sangjin Lee Sangjin Lee
MAPREDUCE-6528 Memory leak for HistoryFileManager.getJobSummary() Critical jobhistoryserver Junping Du Junping Du
MAPREDUCE-6518 Set SO_KEEPALIVE on shuffle connections Major mrv2, nodemanager Nathan Roberts Chang Li
MAPREDUCE-6497 Fix wrong value of JOB_FINISHED event in JobHistoryEventHandler Major . Shinichi Yamashita Shinichi Yamashita
MAPREDUCE-6492 AsyncDispatcher exit with NPE on TaskAttemptImpl#sendJHStartEventForAssignedFailTask Critical . Bibin A Chundatt Bibin A Chundatt
MAPREDUCE-6481 LineRecordReader may give incomplete record and wrong position/key information for uncompressed input sometimes. Critical mrv2 zhihai xu zhihai xu
MAPREDUCE-6474 ShuffleHandler can possibly exhaust nodemanager file descriptors Major mrv2, nodemanager Nathan Roberts Kuhu Shukla
MAPREDUCE-6472 MapReduce AM should have java.io.tmpdir=./tmp to be consistent with tasks Major mr-am Jason Lowe Naganarasimha G R
MAPREDUCE-6454 MapReduce doesn’t set the HADOOP_CLASSPATH for jar lib in distributed cache. Critical . Junping Du Junping Du
MAPREDUCE-6451 DistCp has incorrect chunkFilePath for multiple jobs when strategy is dynamic Major distcp Kuhu Shukla Kuhu Shukla
MAPREDUCE-6442 Stack trace is missing when error occurs in client protocol provider’s constructor Major client Chang Li Chang Li
MAPREDUCE-6439 AM may fail instead of retrying if RM shuts down during the allocate call Critical . Anubhav Dhoot Anubhav Dhoot
MAPREDUCE-6426 TestShuffleHandler#testGetMapOutputInfo is failing Major test Devaraj K zhihai xu
MAPREDUCE-6425 ShuffleHandler passes wrong “base” parameter to getMapOutputInfo if mapId is not in the cache. Major mrv2, nodemanager zhihai xu zhihai xu
MAPREDUCE-6377 JHS sorting on state column not working in webUi Minor jobhistoryserver Bibin A Chundatt zhihai xu
MAPREDUCE-6273 HistoryFileManager should check whether summaryFile exists to avoid FileNotFoundException causing HistoryFileInfo into MOVE_FAILED state Minor jobhistoryserver zhihai xu zhihai xu
MAPREDUCE-5982 Task attempts that fail from the ASSIGNED state can disappear Major mr-am Jason Lowe Chang Li
MAPREDUCE-5948 org.apache.hadoop.mapred.LineRecordReader does not handle multibyte record delimiters well Critical . Kris Geusebroek Akira AJISAKA
MAPREDUCE-5883 “Total megabyte-seconds” in job counters is slightly misleading Minor . Nathan Roberts Nathan Roberts
MAPREDUCE-5649 Reduce cannot use more than 2G memory for the final merge Major mrv2 stanley shi Gera Shegalov
YARN-4434 NodeManager Disk Checker parameter documentation is not correct Minor documentation, nodemanager Takashi Ohnishi Weiwei Yang
YARN-4424 Fix deadlock in RMAppImpl Blocker . Yesha Vora Jian He
YARN-4365 FileSystemNodeLabelStore should check for root dir existence on startup Major resourcemanager Jason Lowe Kuhu Shukla
YARN-4354 Public resource localization fails with NPE Blocker nodemanager Jason Lowe Jason Lowe
YARN-4348 ZKRMStateStore.syncInternal shouldn’t wait for sync completion for avoiding blocking ZK’s event thread Blocker . Tsuyoshi Ozawa Tsuyoshi Ozawa
YARN-4344 NMs reconnecting with changed capabilities can lead to wrong cluster resource calculations Critical resourcemanager Varun Vasudev Varun Vasudev
YARN-4326 Fix TestDistributedShell timeout as AHS in MiniYarnCluster no longer binds to default port 8188 Major . MENG DING MENG DING
YARN-4321 Incessant retries if NoAuthException is thrown by Zookeeper in non HA mode Major resourcemanager Varun Saxena Varun Saxena
YARN-4320 TestJobHistoryEventHandler fails as AHS in MiniYarnCluster no longer binds to default port 8188 Major . Varun Saxena Varun Saxena
YARN-4313 Race condition in MiniMRYarnCluster when getting history server address Major . Jian He Jian He
YARN-4312 TestSubmitApplicationWithRMHA fails on branch-2.7 and branch-2.6 as some of the test cases time out Major . Varun Saxena Varun Saxena
YARN-4281 2.7 RM app page is broken Blocker . Chang Li Chang Li
YARN-4241 Fix typo of property name in yarn-default.xml Major documentation Anthony Rojas Anthony Rojas
YARN-4209 RMStateStore FENCED state doesn’t work due to updateFencedState called by stateMachine.doTransition Critical resourcemanager zhihai xu zhihai xu
YARN-4180 AMLauncher does not retry on failures when talking to NM Critical resourcemanager Anubhav Dhoot Anubhav Dhoot
YARN-4127 RM fail with noAuth error if switched from failover mode to non-failover mode Major resourcemanager Jian He Varun Saxena
YARN-4105 Capacity Scheduler headroom for DRF is wrong Major capacityscheduler Chang Li Chang Li
YARN-4103 RM WebServices missing scheme for appattempts logLinks Major . Jonathan Eagles Jonathan Eagles
YARN-4096 App local logs are leaked if log aggregation fails to initialize for the app Major log-aggregation, nodemanager Jason Lowe Jason Lowe
YARN-4087 Followup fixes after YARN-2019 regarding RM behavior when state-store error occurs Major . Jian He Jian He
YARN-4047 ClientRMService getApplications has high scheduler lock contention Major resourcemanager Jason Lowe Jason Lowe
YARN-4041 Slow delegation token renewal can severely prolong RM recovery Major resourcemanager Jason Lowe Sunil G
YARN-4009 CORS support for ResourceManager REST API Major . Prakash Ramachandran Varun Vasudev
YARN-4005 Completed container whose app is finished is not removed from NMStateStore Major . Jun Gong Jun Gong
YARN-4000 RM crashes with NPE if leaf queue becomes parent queue during restart Major capacityscheduler, resourcemanager Jason Lowe Varun Saxena
YARN-3999 RM hangs on draining events Major . Jian He Jian He
YARN-3990 AsyncDispatcher may overloaded with RMAppNodeUpdateEvent when Node is connected/disconnected Critical resourcemanager Rohith Sharma K S Bibin A Chundatt
YARN-3975 WebAppProxyServlet should not redirect to RM page if AHS is enabled Major . Mit Desai Mit Desai
YARN-3967 Fetch the application report from the AHS if the RM does not know about it Major . Mit Desai Mit Desai
YARN-3925 ContainerLogsUtils#getContainerLogFile fails to read container log files from full disks. Critical nodemanager zhihai xu zhihai xu
YARN-3905 Application History Server UI NPEs when accessing apps run after RM restart Major timelineserver Eric Payne Eric Payne
YARN-3896 RMNode transitioned from RUNNING to REBOOTED because its response id had not been reset synchronously Major resourcemanager Jun Gong Jun Gong
YARN-3878 AsyncDispatcher can hang while stopping if it is configured for draining events on stop Critical . Varun Saxena Varun Saxena
YARN-3857 Memory leak in ResourceManager with SIMPLE mode Critical resourcemanager mujunchao mujunchao
YARN-3802 Two RMNodes for the same NodeId are used in RM sometimes after NM is reconnected. Major resourcemanager zhihai xu zhihai xu
YARN-3798 ZKRMStateStore shouldn’t create new session without occurrance of SESSIONEXPIED Blocker resourcemanager Bibin A Chundatt Varun Saxena
YARN-3793 Several NPEs when deleting local files on NM recovery Major nodemanager Karthik Kambatla Varun Saxena
YARN-3780 Should use equals when compare Resource in RMNodeImpl#ReconnectNodeTransition Minor resourcemanager zhihai xu zhihai xu
YARN-3697 FairScheduler: ContinuousSchedulingThread can fail to shutdown Critical fairscheduler zhihai xu zhihai xu
YARN-3690 [JDK8] ‘mvn site’ fails Major api, site Akira AJISAKA Brahma Reddy Battula
YARN-3624 ApplicationHistoryServer reverses the order of the filters it gets Major timelineserver Mit Desai Mit Desai
YARN-3619 ContainerMetrics unregisters during getMetrics and leads to ConcurrentModificationException Major nodemanager Jason Lowe zhihai xu
YARN-3535 Scheduler must re-request container resources when RMContainer transitions from ALLOCATED to KILLED Critical capacityscheduler, fairscheduler, resourcemanager Peng Zhang Peng Zhang
YARN-3508 Prevent processing preemption events on the main RM dispatcher Major resourcemanager, scheduler Jason Lowe Varun Saxena
YARN-2890 MiniYarnCluster should turn on timeline service if configured to do so Major . Mit Desai Mit Desai
YARN-2859 ApplicationHistoryServer binds to default port 8188 in MiniYARNCluster Critical timelineserver Hitesh Shah Vinod Kumar Vavilapalli
YARN-2019 Retrospect on decision of making RM crashed if any exception throw in ZKRMStateStore Critical . Junping Du Jian He


JIRA Summary Priority Component Reporter Contributor
YARN-3580 [JDK 8] TestClientRMService.testGetLabelsToNodes fails Major test Robert Kanter Robert Kanter


JIRA Summary Priority Component Reporter Contributor
YARN-4101 RM should print alert messages if Zookeeper and Resourcemanager gets connection issue Critical yarn Yesha Vora Xuan Gong
YARN-4092 RM HA UI redirection needs to be fixed when both RMs are in standby mode Major resourcemanager Xuan Gong Xuan Gong
YARN-3969 Allow jobs to be submitted to reservation that is active but does not have any allocations Major capacityscheduler, fairscheduler, resourcemanager Subru Krishnan Subru Krishnan
YARN-3893 Both RM in active state when Admin#transitionToActive failure from refeshAll() Critical resourcemanager Bibin A Chundatt Bibin A Chundatt
YARN-3740 Fixed the typo with the configuration name: APPLICATION_HISTORY_PREFIX_MAX_APPS Major resourcemanager, webapp, yarn Xuan Gong Xuan Gong
YARN-3700 ATS Web Performance issue at load time when large number of jobs Major resourcemanager, webapp, yarn Xuan Gong Xuan Gong
YARN-3136 getTransferredContainers can be a bottleneck during AM registration Major scheduler Jason Lowe Sunil G
YARN-2902 Killing a container that is localizing can orphan resources in the DOWNLOADING state Major nodemanager Jason Lowe Varun Saxena
YARN-2801 Add documentation for node labels feature Major documentation Gururaj Shetty Wangda Tan
YARN-2513 Host framework UIs in YARN for use with the ATS Major timelineserver Jonathan Eagles Jonathan Eagles


JIRA Summary Priority Component Reporter Contributor