Apache Hadoop Changelog

Release 1.3.0 - Unreleased (as of 2018-09-02)

INCOMPATIBLE CHANGES:

JIRA Summary Priority Component Reporter Contributor
MAPREDUCE-5777 Support utf-8 text with BOM (byte order marker) Major . bc Wong zhihai xu

IMPROVEMENTS:

JIRA Summary Priority Component Reporter Contributor
MAPREDUCE-2351 mapred.job.tracker.history.completed.location should support an arbitrary filesystem URI Major . Tom White Tom White
HADOOP-9450 HADOOP_USER_CLASSPATH_FIRST is not honored; CLASSPATH is PREpended instead of APpended Major scripts Mitch Wyle Harsh J
HDFS-4521 invalid network topologies should not be cached Minor . Colin P. McCabe Colin P. McCabe
HADOOP-8873 Port HADOOP-8175 (Add mkdir -p flag) to branch-1 Major . Eli Collins Akira Ajisaka
HDFS-4903 Print trash configuration and trash emptier state in namenode log Minor namenode Suresh Srinivas Arpit Agarwal
MAPREDUCE-5406 Improve logging around Task Tracker exiting with JVM manager inconsistent state Major tasktracker Chelsey Chang Chelsey Chang
MAPREDUCE-5408 CLONE - The logging level of the tasks should be configurable by the job Major . Owen O’Malley Arun C Murthy
MAPREDUCE-5367 Local jobs all use same local working directory Major . Sandy Ryza Sandy Ryza
HDFS-4963 Improve multihoming support in namenode Major namenode Arpit Agarwal Arpit Agarwal
HADOOP-9855 Backport HADOOP-6578 to branch-1 Major . James Kinley James Kinley
HDFS-5038 Backport several branch-2 APIs to branch-1 Minor . Jing Zhao Jing Zhao
HDFS-5367 Restoring namenode storage locks namenode due to unnecessary fsimage write Major . yunjiong zhao yunjiong zhao
MAPREDUCE-5457 Add a KeyOnlyTextOutputReader to enable streaming to write out text files without separators Major . Sandy Ryza Sandy Ryza
MAPREDUCE-5609 Add debug log message when sending job end notification Major . Robert Kanter Robert Kanter
MAPREDUCE-3310 Custom grouping comparator cannot be set for Combiners Major client Mathias Herberts Alejandro Abdelnur
MAPREDUCE-5651 Backport Fair Scheduler queue placement policies to branch-1 Major scheduler Sandy Ryza Ted Malaska
MAPREDUCE-5712 Backport Fair Scheduler pool placement by secondary group Major scheduler Theodore michael Malaska Theodore michael Malaska
HADOOP-10614 CBZip2InputStream is not threadsafe Major . Xiangrui Meng Xiangrui Meng
MAPREDUCE-6088 TestTokenCache tests should use their own JobConf instances Major mrv1, test zhihai xu zhihai xu
HDFS-7312 Update DistCp v1 to optionally not use tmp location (branch-1 only) Minor tools Joseph Prosser Joseph Prosser

BUG FIXES:

JIRA Summary Priority Component Reporter Contributor
HADOOP-7140 IPC Reader threads do not stop when server stops Critical . Todd Lipcon Todd Lipcon
HDFS-2264 NamenodeProtocol has the wrong value for clientPrincipal in KerberosInfo annotation Major namenode Aaron T. Myers Aaron T. Myers
MAPREDUCE-5046 backport MAPREDUCE-1423 to mapred.lib.CombineFileInputFormat Major client Sangjin Lee
MAPREDUCE-5047 keep.failed.task.files=true causes job failure on secure clusters Major task, tasktracker Sandy Ryza Sandy Ryza
MAPREDUCE-5070 TestClusterStatus.testClusterMetrics fails on JDK7 Major test Sandy Ryza Sandy Ryza
MAPREDUCE-5072 TestDelegationTokenRenewal.testDTRenewal fails in MR1 on jdk7 Major test Sandy Ryza Sandy Ryza
MAPREDUCE-5073 TestJobStatusPersistency.testPersistency fails on JDK7 Major test Sandy Ryza Sandy Ryza
HDFS-4622 Remove redundant synchronized from FSNamesystem#rollEditLog in branch-1 Trivial . Jing Zhao Jing Zhao
MAPREDUCE-2817 MiniRMCluster hardcodes ‘mapred.local.dir’ configuration to ‘build/test/mapred/local’ Minor test Alejandro Abdelnur Robert Kanter
MAPREDUCE-5133 TestSubmitJob.testSecureJobExecution is flaky due to job dir deletion race Major test Sandy Ryza Sandy Ryza
MAPREDUCE-5218 Annotate (comment) internal classes as Private Minor . Karthik Kambatla Karthik Kambatla
MAPREDUCE-5217 DistCp fails when launched by Oozie in a secure cluster Major distcp, security Venkat Ranganathan Venkat Ranganathan
MAPREDUCE-5095 TestShuffleExceptionCount#testCheckException fails occasionally with JDK7 Major . Arpit Agarwal Arpit Agarwal
MAPREDUCE-5250 Searching for ‘;’ in JobTracker History throws ArrayOutOfBoundException Minor . Karthik Kambatla Karthik Kambatla
HDFS-4944 WebHDFS cannot create a file path containing characters that must be URI-encoded, such as space. Major webhdfs Chris Nauroth Chris Nauroth
HADOOP-9678 TestRPC#testStopsAllThreads intermittently fails on Windows Major . Ivan Mitic Ivan Mitic
HADOOP-9307 BufferedFSInputStream.read returns wrong results after certain seeks Major fs Todd Lipcon Todd Lipcon
HDFS-5003 TestNNThroughputBenchmark failed caused by existing directories Minor test Xi Fang Xi Fang
MAPREDUCE-5405 Job recovery can fail if task log directory symlink from prior run still exists Major mrv1 Chris Nauroth Chris Nauroth
MAPREDUCE-5288 ResourceEstimator#getEstimatedTotalMapOutputSize suffers from divide by zero issues Major mrv1 Harsh J Karthik Kambatla
HADOOP-9507 LocalFileSystem rename() is broken in some cases when destination exists Minor fs Mostafa Elhemali Chris Nauroth
MAPREDUCE-4366 mapred metrics shows negative count of waiting maps and reduces Major jobtracker Thomas Graves Sandy Ryza
HADOOP-9768 chown and chgrp reject users and groups with spaces on platforms where spaces are otherwise acceptable Major fs Chris Nauroth Chris Nauroth
HADOOP-9801 Configuration#writeXml uses platform defaulting encoding, which may mishandle multi-byte characters. Major conf Chris Nauroth Chris Nauroth
HDFS-4898 BlockPlacementPolicyWithNodeGroup.chooseRemoteRack() fails to properly fallback to local rack Minor namenode Eric Sirianni Tsz Wo Nicholas Sze
HDFS-5211 Race condition between DistributedFileSystem#close and FileSystem#close can cause return of a closed DistributedFileSystem instance from the FileSystem cache. Major hdfs-client Chris Nauroth Chris Nauroth
MAPREDUCE-5508 JobTracker memory leak caused by unreleased FileSystem objects in JobInProgress#cleanupJob Critical jobtracker Xi Fang Xi Fang
HDFS-5245 shouldRetry() in WebHDFSFileSystem generates excessive warnings Minor webhdfs Haohui Mai Haohui Mai
HADOOP-10009 Backport HADOOP-7808 to branch-1 Major . Haohui Mai Haohui Mai
MAPREDUCE-5569 FloatSplitter is not generating correct splits Major . Nathan Roberts Nathan Roberts
MAPREDUCE-5512 TaskTracker hung after failed reconnect to the JobTracker Major tasktracker Ivan Mitic Ivan Mitic
HDFS-4794 Browsing filesystem via webui throws kerberos exception when NN service RPC is enabled in a secure cluster Major security Benoy Antony Benoy Antony
HADOOP-10142 Avoid groups lookup for unprivileged users such as “dr.who” Major . Vinayakumar B Vinayakumar B
HADOOP-10090 Jobtracker metrics not updated properly after execution of a mapreduce job Major metrics Ivan Mitic Ivan Mitic
MAPREDUCE-5702 TaskLogServlet#printTaskLog has spurious HTML closing tags Trivial task Karthik Kambatla Robert Kanter
MAPREDUCE-5698 Backport MAPREDUCE-1285 to branch-1 Major . Yongjun Zhang Yongjun Zhang
HDFS-5685 DistCp will fail to copy with -delete switch Major hdfs-client Yongjun Zhang Yongjun Zhang
MAPREDUCE-5710 Backport MAPREDUCE-1305 to branch-1 Major . Yongjun Zhang Yongjun Zhang
HDFS-5944 LeaseManager:findLeaseWithPrefixPath can’t handle path like /a/b/ right and cause SecondaryNameNode failed do checkpoint Major namenode yunjiong zhao yunjiong zhao
HDFS-5516 WebHDFS does not require user name when anonymous http requests are disallowed. Major webhdfs Chris Nauroth Miodrag Radulovic
MAPREDUCE-5808 Port output replication factor configurable for terasort to Hadoop 1.x Minor examples Chuan Liu Chuan Liu
MAPREDUCE-5822 FairScheduler does not preempt due to fairshare-starvation when fairshare is 1 Major scheduler Anubhav Dhoot Anubhav Dhoot
HDFS-6141 WebHdfsFileSystem#toUrl does not perform character escaping. Major webhdfs Chris Nauroth Chris Nauroth
HADOOP-10562 Namenode exits on exception without printing stack trace in AbstractDelegationTokenSecretManager Critical . Suresh Srinivas Suresh Srinivas
MAPREDUCE-5877 Inconsistency between JT/TT for tasks taking a long time to launch Critical jobtracker, tasktracker Karthik Kambatla Karthik Kambatla
MAPREDUCE-5979 FairScheduler: zero weight can cause sort failures Major scheduler Anubhav Dhoot Anubhav Dhoot
MAPREDUCE-5966 MR1 FairScheduler use of custom weight adjuster is not thread safe for comparisons Major scheduler Anubhav Dhoot Anubhav Dhoot
MAPREDUCE-5968 Work directory is not deleted when downloadCacheObject throws IOException Major mrv1 zhihai xu zhihai xu
HDFS-6822 In branch-1, Namenode and datanode fails to replace “_HOST” to hostname for hadoop.http.authentication.kerberos.principal Major security Jing Zhao Jing Zhao
MAPREDUCE-6012 DBInputSplit creates invalid ranges on Oracle Major . Julien Serdaru Wei Yan
HADOOP-11035 distcp on mr1(branch-1) fails with NPE using a short relative source path. Major tools zhihai xu zhihai xu
MAPREDUCE-6009 Map-only job with new-api runs wrong OutputCommitter when cleanup scheduled in a reduce slot Blocker client, job submission Gera Shegalov Gera Shegalov
MAPREDUCE-6147 Support mapreduce.input.fileinputformat.split.maxsize Minor mrv1 zhihai xu zhihai xu
MAPREDUCE-6170 TestUlimit failure on JDK8 Major contrib/streaming bc Wong bc Wong
HDFS-7503 Namenode restart after large deletions can cause slow processReport (due to logging) Major namenode Arpit Agarwal Arpit Agarwal
MAPREDUCE-6196 Fix BigDecimal ArithmeticException in PiEstimator Minor . Ray Chiang Ray Chiang
MAPREDUCE-6198 NPE from JobTracker#resolveAndAddToTopology in MR1 cause initJob and heartbeat failure. Major mrv1 zhihai xu zhihai xu
MAPREDUCE-5272 A Minor Error in Javadoc of TestMRWithDistributedCache in Branch-1 Trivial test Zhijie Shen Zhijie Shen
MAPREDUCE-5556 mapred docs have incorrect classpath Trivial . Allen Wittenauer Harsh J
MAPREDUCE-5660 Log info about possible thrashing (when using memory-based scheduling in Capacity Scheduler) is not printed Trivial capacity-sched, mrv1, tasktracker Adam Kawa Adam Kawa
MAPREDUCE-5183 In, TaskTracker#reportProgress logging of 0.0-1.0 progress is followed by percent sign Minor mrv1, tasktracker Sandy Ryza Niranjan Singh
HDFS-6649 Documentation for setrep is wrong Trivial documentation Alexander Fahlke Akira Ajisaka
MAPREDUCE-6076 Zero map split input length combine with none zero map split input length may cause MR1 job hung sometimes. Major mrv1 zhihai xu zhihai xu

TESTS:

JIRA Summary Priority Component Reporter Contributor
HADOOP-9624 TestFSMainOperationsLocalFileSystem failed when the Hadoop test root path has “X” in its name Minor test Xi Fang Xi Fang

SUB-TASKS:

JIRA Summary Priority Component Reporter Contributor
MAPREDUCE-4049 plugin for generic shuffle service Major performance, task, tasktracker Avner BenHanoch Avner BenHanoch
HADOOP-9573 Fix test-patch script to work with the enhanced PreCommit-Admin script. Major build Giridharan Kesavan Giridharan Kesavan