Apache Hadoop Changelog

Release 2.6.1 - 2015-09-23

INCOMPATIBLE CHANGES:

JIRA Summary Priority Component Reporter Contributor

IMPORTANT ISSUES:

JIRA Summary Priority Component Reporter Contributor

NEW FEATURES:

JIRA Summary Priority Component Reporter Contributor

IMPROVEMENTS:

JIRA Summary Priority Component Reporter Contributor
HADOOP-12280 Skip unit tests based on maven profile rather than NativeCodeLoader.isNativeCodeLoaded Minor test Masatake Iwasaki Masatake Iwasaki
HADOOP-11812 Implement listLocatedStatus for ViewFileSystem to speed up split calculation Blocker fs Gera Shegalov Gera Shegalov
HADOOP-11506 Configuration variable expansion regex expensive for long values Major conf Dmitriy V. Ryaboy Gera Shegalov
HADOOP-11466 FastByteComparisons: do not use UNSAFE_COMPARER on the SPARC architecture because it is slower there Minor io, performance, util Suman Somasundar Suman Somasundar
HADOOP-7139 Allow appending to existing SequenceFiles Major io Stephen Rose Kanaka Kumar Avvaru
HDFS-8384 Allow NN to startup if there are files having a lease but are not under construction Minor namenode Tsz Wo Nicholas Sze Jing Zhao
HDFS-7579 Improve log reporting during block report rpc failure Minor datanode Charles Lamb Charles Lamb
HDFS-7531 Improve the concurrent access on FsVolumeList Major datanode Lei (Eddy) Xu Lei (Eddy) Xu
HDFS-7446 HDFS inotify should have the ability to determine what txid it has read up to Major hdfs-client Colin Patrick McCabe Colin Patrick McCabe
HDFS-7314 When the DFSClient lease cannot be renewed, abort open-for-write files rather than the entire DFSClient Major . Ming Ma Ming Ma
HDFS-7278 Add a command that allows sysadmins to manually trigger full block reports from a DN Major datanode Colin Patrick McCabe Colin Patrick McCabe
HDFS-7182 JMX metrics aren’t accessible when NN is busy Major . Ming Ma Ming Ma
MAPREDUCE-6267 Refactor JobSubmitter#copyAndConfigureFiles into it’s own class Minor . Chris Trezzo Chris Trezzo
YARN-3978 Configurably turn off the saving of container info in Generic AHS Major timelineserver, yarn Eric Payne Eric Payne
YARN-3249 Add a “kill application” button to Resource Manager’s Web UI Minor resourcemanager Ryu Kobayashi Ryu Kobayashi
YARN-3248 Display count of nodes blacklisted by apps in the web UI Major capacityscheduler, resourcemanager Varun Vasudev Varun Vasudev
YARN-3230 Clarify application states on the web UI Major . Jian He Jian He
YARN-2301 Improve yarn container command Major . Jian He Naganarasimha G R

BUG FIXES:

JIRA Summary Priority Component Reporter Contributor
HADOOP-11934 Use of JavaKeyStoreProvider in LdapGroupsMapping causes infinite loop Blocker security Mike Yoder Larry McCay
HADOOP-11932 MetricsSinkAdapter hangs when being stopped Critical . Jian He Brahma Reddy Battula
HADOOP-11802 DomainSocketWatcher thread terminates sometimes after there is an I/O error during requestShortCircuitShm Major . Eric Payne Colin Patrick McCabe
HADOOP-11730 Regression: s3n read failure recovery broken Major fs/s3 Takenori Sato Takenori Sato
HADOOP-11674 oneByteBuf in CryptoInputStream and CryptoOutputStream should be non static Critical io Sean Busbey Sean Busbey
HADOOP-11604 Prevent ConcurrentModificationException while closing domain sockets during shutdown of DomainSocketWatcher thread. Critical net Liang Xie Chris Nauroth
HADOOP-11491 HarFs incorrectly declared as requiring an authority Critical fs Gera Shegalov Brahma Reddy Battula
HADOOP-11482 Use correct UGI when KMSClientProvider is called by a proxy user Major . Arun Suresh Arun Suresh
HADOOP-11368 Fix SSLFactory truststore reloader thread leak in KMSClientProvider Major kms Arun Suresh Arun Suresh
HADOOP-11350 The size of header buffer of HttpServer is too small when HTTPS is enabled Major security Benoy Antony Benoy Antony
HADOOP-11343 Overflow is not properly handled in caclulating final iv for AES CTR Blocker security Jerry Chen Jerry Chen
HADOOP-11333 Fix deadlock in DomainSocketWatcher when the notification pipe is full Major . zhaoyunjiong zhaoyunjiong
HADOOP-11316 “mvn package -Pdist,docs -DskipTests -Dtar” fails because of non-ascii characters Blocker . Tsuyoshi Ozawa Tsuyoshi Ozawa
HADOOP-11295 RPC Server Reader thread can’t shutdown if RPCCallQueue is full Major . Ming Ma Ming Ma
HADOOP-11238 Update the NameNode’s Group Cache in the background when possible Minor . Chris Li Chris Li
HADOOP-10786 Fix UGI#reloginFromKeytab on Java 8 Major security Tobi Vollebregt Stephen Chu
HADOOP-8151 Error handling in snappy decompressor throws invalid exceptions Major io, native Todd Lipcon Matt Foley
HDFS-8863 The remaining space check in BlockPlacementPolicyDefault is flawed Critical . Kihwal Lee Kihwal Lee
HDFS-8846 Add a unit test for INotify functionality across a layout version upgrade Major namenode Zhe Zhang Zhe Zhang
HDFS-8486 DN startup may cause severe data loss Blocker datanode Daryn Sharp Daryn Sharp
HDFS-8480 Fix performance and timeout issues in HDFS-7929 by using hard-links to preserve old edit logs instead of copying them Critical . Zhe Zhang Zhe Zhang
HDFS-8431 hdfs crypto class not found in Windows Critical scripts Sumana Sathish Anu Engineer
HDFS-8404 Pending block replication can get stuck using older genstamp Major namenode Nathan Roberts Nathan Roberts
HDFS-8270 create() always retried with hardcoded timeout when file already exists with open lease Major hdfs-client Andrey Stepachev J.Andreina
HDFS-8245 Standby namenode doesn’t process DELETED_BLOCK if the add block request is in edit log. Major . Rushabh S Shah Rushabh S Shah
HDFS-8219 setStoragePolicy with folder behavior is different after cluster restart Major . Peter Shi Surendra Singh Lilhore
HDFS-8127 NameNode Failover during HA upgrade can cause DataNode to finalize upgrade Blocker ha Jing Zhao Jing Zhao
HDFS-8072 Reserved RBW space is not released if client terminates while writing block Major datanode Arpit Agarwal Arpit Agarwal
HDFS-8070 Pre-HDFS-7915 DFSClient cannot use short circuit on post-HDFS-7915 DataNode Blocker caching Gopal V Colin Patrick McCabe
HDFS-8046 Allow better control of getContentSummary Major . Kihwal Lee Kihwal Lee
HDFS-7999 FsDatasetImpl#createTemporary sometimes holds the FSDatasetImpl lock for a very long time Major . zhouyingchao zhouyingchao
HDFS-7980 Incremental BlockReport will dramatically slow down the startup of a namenode Major . Hui Zheng Walter Su
HDFS-7960 The full block report should prune zombie storages even if they’re not empty Critical . Lei (Eddy) Xu Colin Patrick McCabe
HDFS-7930 commitBlockSynchronization() does not remove locations Blocker namenode Konstantin Shvachko Yi Liu
HDFS-7929 inotify unable fetch pre-upgrade edit log segments once upgrade starts Major . Zhe Zhang Zhe Zhang
HDFS-7915 The DataNode can sometimes allocate a ShortCircuitShm slot and fail to tell the DFSClient about it because of a network error Major . Colin Patrick McCabe Colin Patrick McCabe
HDFS-7894 Rolling upgrade readiness is not updated in jmx until query command is issued. Critical . Kihwal Lee Brahma Reddy Battula
HDFS-7885 Datanode should not trust the generation stamp provided by client Critical datanode vitthal (Suhas) Gogate Tsz Wo Nicholas Sze
HDFS-7884 NullPointerException in BlockSender Blocker datanode Tsz Wo Nicholas Sze Brahma Reddy Battula
HDFS-7871 NameNodeEditLogRoller can keep printing “Swallowing exception” message Critical . Jing Zhao Jing Zhao
HDFS-7830 DataNode does not release the volume lock when adding a volume fails. Major datanode Lei (Eddy) Xu Lei (Eddy) Xu
HDFS-7788 Post-2.6 namenode may not start up with an image containing inodes created with an old release. Blocker . Kihwal Lee Rushabh S Shah
HDFS-7763 fix zkfc hung issue due to not catching exception in a corner case Major ha Liang Xie Liang Xie
HDFS-7742 favoring decommissioning node for replication can cause a block to stay underreplicated for long periods Major namenode Nathan Roberts Nathan Roberts
HDFS-7733 NFS: readdir/readdirplus return null directory attribute on failure Major nfs Arpit Agarwal Arpit Agarwal
HDFS-7714 Simultaneous restart of HA NameNodes and DataNode can cause DataNode to register successfully with only one NameNode. Major datanode Chris Nauroth Vinayakumar B
HDFS-7707 Edit log corruption due to delayed block removal again Major namenode Yongjun Zhang Yongjun Zhang
HDFS-7610 Fix removal of dynamically added DN volumes Major datanode Lei (Eddy) Xu Lei (Eddy) Xu
HDFS-7609 Avoid retry cache collision when Standby NameNode loading edits Critical namenode Carrey Zhan Ming Ma
HDFS-7596 NameNode should prune dead storages from storageMap Major namenode Arpit Agarwal Arpit Agarwal
HDFS-7587 Edit log corruption can happen if append fails with a quota violation Blocker namenode Kihwal Lee Jing Zhao
HDFS-7575 Upgrade should generate a unique storage ID for each volume Critical . Lars Francke Arpit Agarwal
HDFS-7552 change FsVolumeList toString() to fix TestDataNodeVolumeFailureToleration Major datanode, test Liang Xie Liang Xie
HDFS-7533 Datanode sometimes does not shutdown on receiving upgrade shutdown command Major . Kihwal Lee Eric Payne
HDFS-7503 Namenode restart after large deletions can cause slow processReport (due to logging) Major namenode Arpit Agarwal Arpit Agarwal
HDFS-7489 Incorrect locking in FsVolumeList#checkDirs can hang datanodes Critical datanode Noah Lorang Noah Lorang
HDFS-7470 SecondaryNameNode need twice memory when calling reloadFromImageFile Major namenode zhaoyunjiong zhaoyunjiong
HDFS-7443 Datanode upgrade to BLOCKID_BASED_LAYOUT fails if duplicate block files are present in the same volume Blocker . Kihwal Lee Colin Patrick McCabe
HDFS-7425 NameNode block deletion logging uses incorrect appender. Minor namenode Chris Nauroth Chris Nauroth
HDFS-7263 Snapshot read can reveal future bytes for appended files. Major hdfs-client Konstantin Shvachko Tao Luo
HDFS-7235 DataNode#transferBlock should report blocks that don’t exist using reportBadBlock Major datanode, namenode Yongjun Zhang Yongjun Zhang
HDFS-7225 Remove stale block invalidation work when DN re-registers with different UUID Major namenode Zhe Zhang Zhe Zhang
HDFS-7213 processIncrementalBlockReport performance degradation Critical namenode Daryn Sharp Eric Payne
HDFS-7009 Active NN and standby NN have different live nodes Major datanode Ming Ma Ming Ma
HDFS-4882 Prevent the Namenode’s LeaseManager from looping forever in checkLeases Critical hdfs-client, namenode Zesheng Wu Ravi Prakash
HDFS-3443 Fix NPE when namenode transition to active during startup by adding checkNNStartup() in NameNodeRpcServer Major auto-failover, ha suja s Vinayakumar B
MAPREDUCE-6361 NPE issue in shuffle caused by concurrent issue between copySucceeded() in one thread and copyFailed() in another thread on the same host Critical . Junping Du Junping Du
MAPREDUCE-6324 Uber jobs fail to update AMRM token when it rolls over Blocker mr-am Jason Lowe Jason Lowe
MAPREDUCE-6303 Read timeout when retrying a fetch error can be fatal to a reducer Blocker . Jason Lowe Jason Lowe
MAPREDUCE-6300 Task list sort by task id broken Minor . Siqi Li Siqi Li
MAPREDUCE-6238 MR2 can’t run local jobs with -libjars command options which is a regression from MR1 Critical mrv2 zhihai xu zhihai xu
MAPREDUCE-6237 Multiple mappers with DBInputFormat don’t work because of reusing conections Major mrv2 Kannan Rajah Kannan Rajah
MAPREDUCE-6230 MR AM does not survive RM restart if RM activated a new AMRM secret key Blocker mr-am Jason Lowe Jason Lowe
MAPREDUCE-6166 Reducers do not validate checksum of map outputs when fetching directly to disk Major mrv2 Eric Payne Eric Payne
MAPREDUCE-5649 Reduce cannot use more than 2G memory for the final merge Major mrv2 stanley shi Gera Shegalov
YARN-4047 ClientRMService getApplications has high scheduler lock contention Major resourcemanager Jason Lowe Jason Lowe
YARN-3999 RM hangs on draining events Major . Jian He Jian He
YARN-3990 AsyncDispatcher may overloaded with RMAppNodeUpdateEvent when Node is connected/disconnected Critical resourcemanager Rohith Sharma K S Bibin A Chundatt
YARN-3850 NM fails to read files from full disks which can lead to container logs being lost and other issues Blocker log-aggregation, nodemanager Varun Saxena Varun Saxena
YARN-3832 Resource Localization fails on a cluster due to existing cache directories Critical nodemanager Ranga Swamy Brahma Reddy Battula
YARN-3733 Fix DominantRC#compare() does not work as expected if cluster resource is empty Blocker resourcemanager Bibin A Chundatt Rohith Sharma K S
YARN-3725 App submission via REST API is broken in secure mode due to Timeline DT service address is empty Blocker resourcemanager, timelineserver Zhijie Shen Zhijie Shen
YARN-3585 NodeManager cannot exit on SHUTDOWN event triggered and NM recovery is enabled Critical . Peng Zhang Rohith Sharma K S
YARN-3526 ApplicationMaster tracking URL is incorrectly redirected on a QJM cluster Major resourcemanager, webapp Weiwei Yang Weiwei Yang
YARN-3493 RM fails to come up with error “Failed to load/recover state” when mem settings are changed Critical yarn Sumana Sathish Jian He
YARN-3464 Race condition in LocalizerRunner kills localizer before localizing all resources Critical nodemanager zhihai xu zhihai xu
YARN-3393 Getting application(s) goes wrong when app finishes before starting the attempt Critical timelineserver Zhijie Shen Zhijie Shen
YARN-3369 Missing NullPointer check in AppSchedulingInfo causes RM to die Blocker resourcemanager Giovanni Matteo Fumarola Brahma Reddy Battula
YARN-3287 TimelineClient kerberos authentication failure uses wrong login context. Major . Jonathan Eagles Daryn Sharp
YARN-3267 Timelineserver applies the ACL rules after applying the limit on the number of records Major . Prakash Ramachandran Chang Li
YARN-3251 Fix CapacityScheduler deadlock when computing absolute max avail capacity (short term fix for 2.6.1) Blocker . Jason Lowe Craig Welch
YARN-3242 Asynchrony in ZK-close can lead to ZKRMStateStore watcher receiving events for old client Critical resourcemanager zhihai xu zhihai xu
YARN-3239 WebAppProxy does not support a final tracking url which has query fragments and params Major . Hitesh Shah Jian He
YARN-3238 Connection timeouts to nodemanagers are retried at multiple levels Blocker . Jason Lowe Jason Lowe
YARN-3231 FairScheduler: Changing queueMaxRunningApps interferes with pending jobs Critical . Siqi Li Siqi Li
YARN-3227 Timeline renew delegation token fails when RM user’s TGT is expired Critical . Jonathan Eagles Zhijie Shen
YARN-3222 RMNodeImpl#ReconnectNodeTransition should send scheduler events in sequential order Critical resourcemanager Rohith Sharma K S Rohith Sharma K S
YARN-3207 secondary filter matches entites which do not have the key being filtered for. Major timelineserver Prakash Ramachandran Zhijie Shen
YARN-3103 AMRMClientImpl does not update AMRM token properly Blocker client Jason Lowe Jason Lowe
YARN-3094 reset timer for liveness monitors after RM recovery Major resourcemanager Jun Gong Jun Gong
YARN-3055 The token is not renewed properly if it’s shared by jobs (oozie) in DelegationTokenRenewer Blocker security Yi Liu Daryn Sharp
YARN-3024 LocalizerRunner should give DIE action when all resources are localized Major nodemanager Chengbing Liu Chengbing Liu
YARN-2997 NM keeps sending already-sent completed containers to RM until containers are removed from context Major nodemanager Chengbing Liu Chengbing Liu
YARN-2992 ZKRMStateStore crashes due to session expiry Blocker resourcemanager Karthik Kambatla Karthik Kambatla
YARN-2978 ResourceManager crashes with NPE while getting queue info Critical . Jason Tufo Varun Saxena
YARN-2964 RM prematurely cancels tokens for jobs that submit jobs (oozie) Blocker resourcemanager Daryn Sharp Jian He
YARN-2952 Incorrect version check in RMStateStore Major . Jian He Rohith Sharma K S
YARN-2922 ConcurrentModificationException in CapacityScheduler’s LeafQueue Major capacityscheduler, resourcemanager, scheduler Jason Tufo Rohith Sharma K S
YARN-2917 Potential deadlock in AsyncDispatcher when system.exit called in AsyncDispatcher#dispatch and AsyscDispatcher#serviceStop from shutdown hook Critical resourcemanager Rohith Sharma K S Rohith Sharma K S
YARN-2910 FSLeafQueue can throw ConcurrentModificationException Major fairscheduler Wilfred Spiegelenburg Wilfred Spiegelenburg
YARN-2906 CapacitySchedulerPage shows HTML tags for a queue’s Active Users Major capacityscheduler Jason Lowe Jason Lowe
YARN-2905 AggregatedLogsBlock page can infinitely loop if the aggregated log file is corrupted Blocker . Jason Lowe Varun Saxena
YARN-2894 When ACL’s are enabled, if RM switches then application can not be viewed from web. Major resourcemanager Rohith Sharma K S Rohith Sharma K S
YARN-2890 MiniYarnCluster should turn on timeline service if configured to do so Major . Mit Desai Mit Desai
YARN-2874 Dead lock in “DelegationTokenRenewer” which blocks RM to execute any further apps Blocker resourcemanager Naganarasimha G R Naganarasimha G R
YARN-2865 Application recovery continuously fails with “Application with id already present. Cannot duplicate” Critical resourcemanager Rohith Sharma K S Rohith Sharma K S
YARN-2856 Application recovery throw InvalidStateTransitonException: Invalid event: ATTEMPT_KILLED at ACCEPTED Critical resourcemanager Rohith Sharma K S Rohith Sharma K S
YARN-2816 NM fail to start with NPE during container recovery Major nodemanager zhihai xu zhihai xu
YARN-2637 maximum-am-resource-percent could be respected for both LeafQueue/User when trying to activate applications. Critical resourcemanager Wangda Tan Craig Welch
YARN-2414 RM web UI: app page will crash if app is failed before any attempt has been created Major webapp Zhijie Shen Wangda Tan
YARN-2340 NPE thrown when RM restart after queue is STOPPED. There after RM can not recovery application’s and remain in standby Critical resourcemanager, scheduler Nishan Shetty Rohith Sharma K S
YARN-2246 Job History Link in RM UI is redirecting to the URL which contains Job Id twice Major webapp Devaraj K Devaraj K

TESTS:

JIRA Summary Priority Component Reporter Contributor

SUB-TASKS:

JIRA Summary Priority Component Reporter Contributor
HADOOP-11710 Make CryptoOutputStream behave like DFSOutputStream wrt synchronization Critical fs Sean Busbey Sean Busbey
HDFS-7035 Make adding a new data directory to the DataNode an atomic operation and improve error handling Major datanode Lei (Eddy) Xu Lei (Eddy) Xu
YARN-3740 Fixed the typo with the configuration name: APPLICATION_HISTORY_PREFIX_MAX_APPS Major resourcemanager, webapp, yarn Xuan Gong Xuan Gong
YARN-3700 ATS Web Performance issue at load time when large number of jobs Major resourcemanager, webapp, yarn Xuan Gong Xuan Gong
YARN-3544 AM logs link missing in the RM UI for a completed app Blocker . Hitesh Shah Xuan Gong
YARN-3487 CapacityScheduler scheduler lock obtained unnecessarily when calling getQueue Critical capacityscheduler Jason Lowe Jason Lowe
YARN-3171 Sort by Application id, AppAttempt & ContainerID doesn’t work in ATS / RM web ui Minor timelineserver Jeff Zhang Naganarasimha G R
YARN-3124 Capacity Scheduler LeafQueue/ParentQueue should use QueueCapacities to track capacities-by-label Major api, client, resourcemanager Wangda Tan Wangda Tan
YARN-3099 Capacity Scheduler LeafQueue/ParentQueue should use ResourceUsage to track used-resources-by-label. Major api, client, resourcemanager Wangda Tan Wangda Tan
YARN-3098 Create common QueueCapacities class in Capacity Scheduler to track capacities-by-labels of queues Major capacityscheduler Wangda Tan Wangda Tan
YARN-3092 Create common ResourceUsage class to track labeled resource usages in Capacity Scheduler Major api, client, resourcemanager Wangda Tan Wangda Tan
YARN-3011 NM dies because of the failure of resource localization Major nodemanager Wang Hao Varun Saxena
YARN-2920 CapacityScheduler should be notified when labels on nodes changed Major . Wangda Tan Wangda Tan
YARN-2918 Don’t fail RM if queue’s configured labels are not existed in cluster-node-labels Major resourcemanager Rohith Sharma K S Wangda Tan
YARN-2900 Application (Attempt and Container) Not Found in AHS results in Internal Server Error (500) Major timelineserver Jonathan Eagles Mit Desai
YARN-2766 ApplicationHistoryManager is expected to return a sorted list of apps/attempts/containers Major timelineserver Robert Kanter Robert Kanter
YARN-2694 Ensure only single node labels specified in resource request / host, and node label expression only specified when resourceName=ANY Major capacityscheduler, resourcemanager Wangda Tan Wangda Tan
YARN-1984 LeveldbTimelineStore does not handle db exceptions properly Major . Jason Lowe Varun Saxena
YARN-1884 ContainerReport should have nodeHttpAddress Major . Zhijie Shen Xuan Gong
YARN-1809 Synchronize RM and Generic History Service Web-UIs Major . Zhijie Shen Xuan Gong

OTHER:

JIRA Summary Priority Component Reporter Contributor