Apache Hadoop Changelog

Release 0.23.5 - 2012-11-29

INCOMPATIBLE CHANGES:

JIRA Summary Priority Component Reporter Contributor
HDFS-4080 Add a separate logger for block state change logs to enable turning off those logs Major namenode Kihwal Lee Kihwal Lee

IMPROVEMENTS:

JIRA Summary Priority Component Reporter Contributor
HADOOP-8789 Tests setLevel(Level.OFF) should be Level.ERROR Minor test Andy Isaacson Andy Isaacson
HADOOP-8755 Print thread dump when tests fail due to timeout Major test Andrey Klochkov Andrey Klochkov
HADOOP-8851 Use -XX:+HeapDumpOnOutOfMemoryError JVM option in the forked tests Minor test Ivan A. Veselovsky Ivan A. Veselovsky
HDFS-3483 Better error message when hdfs fsck is run against a ViewFS config Major . Stephen Chu Stephen Fritz
HADOOP-8889 Upgrade to Surefire 2.12.3 Major build, test Todd Lipcon Todd Lipcon
HADOOP-8932 JNI-based user-group mapping modules can be too chatty on lookup failures Major security Kihwal Lee Kihwal Lee
HADOOP-8926 hadoop.util.PureJavaCrc32 cache hit-ratio is low for static data Major util Gopal V Gopal V
HADOOP-8930 Cumulative code coverage calculation Major test Andrey Klochkov Andrey Klochkov
MAPREDUCE-4229 Counter names’ memory usage can be decreased by interning Major jobtracker Todd Lipcon Miomir Boljanovic
MAPREDUCE-4752 Reduce MR AM memory usage through String Interning Major mrv2 Robert Joseph Evans Robert Joseph Evans
YARN-165 RM should point tracking URL to RM web page for app when AM fails Blocker resourcemanager Jason Lowe Jason Lowe
MAPREDUCE-4763 repair test org.apache.hadoop.mapreduce.security.TestUmbilicalProtocolWithJobToken Minor . Ivan A. Veselovsky
MAPREDUCE-4666 JVM metrics for history server Minor jobhistoryserver Jason Lowe Jason Lowe
MAPREDUCE-4517 Too many INFO messages written out during AM to RM heartbeat Minor applicationmaster James Kinley Jason Lowe
YARN-216 Remove jquery theming support Major . Todd Lipcon Robert Joseph Evans
MAPREDUCE-4802 Takes a long time to load the task list on the AM for large jobs Major mr-am, mrv2, webapps Ravi Prakash Ravi Prakash

BUG FIXES:

JIRA Summary Priority Component Reporter Contributor
HADOOP-8775 MR2 distcp permits non-positive value to -bandwidth option which causes job never to complete Major . Sandy Ryza Sandy Ryza
HDFS-3919 MiniDFSCluster:waitClusterUp can hang forever Minor test Andy Isaacson Andy Isaacson
HADOOP-8819 Should use && instead of & in a few places in FTPFileSystem,FTPInputStream,S3InputStream,ViewFileSystem,ViewFs Major fs Brandon Li Brandon Li
YARN-28 TestCompositeService fails on jdk7 Major . Thomas Graves Thomas Graves
MAPREDUCE-4674 Hadoop examples secondarysort has a typo “secondarysrot” in the usage Minor . Robert Justice Robert Justice
HADOOP-8791 rm “Only deletes non empty directory and files.” Major documentation Bertrand Dechoux Jing Zhao
HADOOP-8386 hadoop script doesn’t work if ‘cd’ prints to stdout (default behavior in Ubuntu) Major scripts Christopher Berner Christopher Berner
YARN-116 RM is missing ability to add include/exclude files without a restart Major resourcemanager xieguiming xieguiming
YARN-131 Incorrect ACL properties in capacity scheduler documentation Major capacityscheduler Ahmed Radwan Ahmed Radwan
HDFS-3905 Secure cluster cannot use hftp to an insecure cluster Critical hdfs-client, security Daryn Sharp Daryn Sharp
YARN-102 Move the apache licence header to the top of the file in MemStore.java Trivial resourcemanager Devaraj K Devaraj K
HDFS-3996 Add debug log removed in HDFS-3873 back Minor . Eli Collins Eli Collins
HDFS-3829 TestHftpURLTimeouts fails intermittently with JDK7 Major test Trevor Robinson Trevor Robinson
MAPREDUCE-4554 Job Credentials are not transmitted if security is turned off Major job submission, security Benoy Antony Benoy Antony
MAPREDUCE-4705 Historyserver links expire before the history data does Critical jobhistoryserver, mrv2 Jason Lowe Jason Lowe
YARN-30 TestNMWebServicesApps, TestRMWebServicesApps and TestRMWebServicesNodes fail on jdk7 Major . Thomas Graves Thomas Graves
HDFS-3824 TestHftpDelegationToken fails intermittently with JDK7 Major test Trevor Robinson Trevor Robinson
HDFS-3224 Bug in check for DN re-registration with different storage ID Minor . Eli Collins Jason Lowe
HADOOP-8906 paths with multiple globs are unreliable Critical fs Daryn Sharp Daryn Sharp
HDFS-4016 back-port HDFS-3582 to branch-0.23 Minor . Ivan A. Veselovsky Ivan A. Veselovsky
MAPREDUCE-4521 mapreduce.user.classpath.first incompatibility with 0.20/1.x Major mrv2 Jason Lowe Ravi Prakash
YARN-161 Yarn Common has multiple compiler warnings for unchecked operations Major api Chris Nauroth Chris Nauroth
YARN-43 TestResourceTrackerService fail intermittently on jdk7 Major . Thomas Graves Thomas Graves
MAPREDUCE-4721 Task startup time in JHS is same as job startup time. Major jobhistoryserver Ravi Prakash Ravi Prakash
YARN-32 TestApplicationTokens fails intermintently on jdk7 Major . Thomas Graves Vinod Kumar Vavilapalli
YARN-163 Retrieving container log via NM webapp can hang with multibyte characters in log Major nodemanager Jason Lowe Jason Lowe
MAPREDUCE-4479 Fix parameter order in assertEquals() in TestCombineInputFileFormat.java Major test Mariappan Asokan Mariappan Asokan
MAPREDUCE-4733 Reducer can fail to make progress during shuffle if too many reducers complete consecutively Major applicationmaster, mrv2 Jason Lowe Jason Lowe
MAPREDUCE-4740 only .jars can be added to the Distributed Cache classpath Blocker mrv2 Robert Joseph Evans Robert Joseph Evans
YARN-174 TestNodeStatusUpdater is failing in trunk Major nodemanager Robert Joseph Evans Vinod Kumar Vavilapalli
HDFS-4090 getFileChecksum() result incompatible when called against zero-byte files. Critical hdfs-client Kihwal Lee Kihwal Lee
HADOOP-8962 RawLocalFileSystem.listStatus fails when a child filename contains a colon Critical fs Jason Lowe Jason Lowe
YARN-177 CapacityScheduler - adding a queue while the RM is running has wacky results Critical capacityscheduler Thomas Graves Arun C Murthy
YARN-178 Fix custom ProcessTree instance creation Critical . Radim Kolar Radim Kolar
YARN-180 Capacity scheduler - containers that get reserved create container token to early Critical capacityscheduler Thomas Graves Arun C Murthy
YARN-139 Interrupted Exception within AsyncDispatcher leads to user confusion Major api Nathan Roberts Vinod Kumar Vavilapalli
MAPREDUCE-4741 WARN and ERROR messages logged during normal AM shutdown Minor applicationmaster, mrv2 Jason Lowe Vinod Kumar Vavilapalli
MAPREDUCE-4730 AM crashes due to OOM while serving up map task completion events Blocker applicationmaster, mrv2 Jason Lowe Jason Lowe
MAPREDUCE-4748 Invalid event: T_ATTEMPT_SUCCEEDED at SUCCEEDED Blocker mrv2 Robert Joseph Evans Jason Lowe
MAPREDUCE-1806 CombineFileInputFormat does not work with paths not on default FS Major harchive Paul Yang Gera Shegalov
HADOOP-8986 Server$Call object is never released after it is sent Critical ipc Robert Joseph Evans Robert Joseph Evans
YARN-159 RM web ui applications page should be sorted to display last app first Major resourcemanager Thomas Graves Thomas Graves
YARN-166 capacity scheduler doesn’t allow capacity < 1.0 Major capacityscheduler Thomas Graves Thomas Graves
MAPREDUCE-4724 job history web ui applications page should be sorted to display last app first Major jobhistoryserver Thomas Graves Thomas Graves
YARN-189 deadlock in RM - AMResponse object Blocker resourcemanager Thomas Graves Thomas Graves
MAPREDUCE-4746 The MR Application Master does not have a config to set environment variables Major applicationmaster Robert Parker Robert Parker
MAPREDUCE-4729 job history UI not showing all job attempts Major jobhistoryserver Thomas Graves Vinod Kumar Vavilapalli
MAPREDUCE-4771 KeyFieldBasedPartitioner not partitioning properly when configured Major mrv2 Jason Lowe Jason Lowe
HDFS-4075 Reduce recommissioning overhead Critical namenode Kihwal Lee Kihwal Lee
YARN-201 CapacityScheduler can take a very long time to schedule containers if requests are off cluster Critical capacityscheduler Jason Lowe Jason Lowe
MAPREDUCE-4772 Fetch failures can take way too long for a map to be restarted Critical mrv2 Robert Joseph Evans Robert Joseph Evans
HDFS-3990 NN’s health report has severe performance problems Critical namenode Daryn Sharp Daryn Sharp
MAPREDUCE-4782 NLineInputFormat skips first line of last InputSplit Blocker client Mark Fuhs Mark Fuhs
HDFS-4162 Some malformed and unquoted HTML strings are returned from datanode web ui Minor datanode Derek Dagit Derek Dagit
MAPREDUCE-4774 JobImpl does not handle asynchronous task events in FAILED state Major applicationmaster, mrv2 Ivan A. Veselovsky Jason Lowe
YARN-206 TestApplicationCleanup.testContainerCleanup occasionally fails Major resourcemanager Jason Lowe Jason Lowe
MAPREDUCE-4751 AM stuck in KILL_WAIT for days Major . Ravi Prakash Vinod Kumar Vavilapalli
MAPREDUCE-4787 TestJobMonitorAndPrint is broken Major test Ravi Prakash Robert Parker
HDFS-4172 namenode does not URI-encode parameters when building URI for datanode request Minor namenode Derek Dagit Derek Dagit
MAPREDUCE-4425 Speculation + Fetch failures can lead to a hung job Critical mrv2 Siddharth Seth Jason Lowe
MAPREDUCE-4786 Job End Notification retry interval is 5 milliseconds by default Major mrv2 Ravi Prakash Ravi Prakash
HADOOP-9022 Hadoop distcp tool fails to copy file if -m 0 specified Major . Haiyang Jiang Jonathan Eagles
HADOOP-9025 org.apache.hadoop.tools.TestCopyListing failing Major . Robert Joseph Evans Jonathan Eagles
HDFS-4181 LeaseManager tries to double remove and prints extra messages Critical namenode Kihwal Lee Kihwal Lee
YARN-212 NM state machine ignores an APPLICATION_CONTAINER_FINISHED event when it shouldn’t Blocker nodemanager Nathan Roberts Nathan Roberts
YARN-144 MiniMRYarnCluster launches RM and JHS on default ports Major . Robert Parker Robert Parker
HDFS-4182 SecondaryNameNode leaks NameCache entries Critical namenode Todd Lipcon Robert Joseph Evans
MAPREDUCE-4797 LocalContainerAllocator can loop forever trying to contact the RM Major applicationmaster Jason Lowe Jason Lowe
MAPREDUCE-4720 Browser thinks History Server main page JS is taking too long Major . Robert Joseph Evans Ravi Prakash
HDFS-4186 logSync() is called with the write lock held while releasing lease Critical namenode Kihwal Lee Kihwal Lee
MAPREDUCE-4801 ShuffleHandler can generate large logs due to prematurely closed channels Critical . Jason Lowe Jason Lowe
YARN-214 RMContainerImpl does not handle event EXPIRE at state RUNNING Major resourcemanager Jason Lowe Jonathan Eagles
YARN-151 Browser thinks RM main page JS is taking too long Major . Robert Joseph Evans Ravi Prakash
MAPREDUCE-4549 Distributed cache conflicts breaks backwards compatability Blocker mrv2 Robert Joseph Evans Robert Joseph Evans

SUB-TASKS:

JIRA Summary Priority Component Reporter Contributor
MAPREDUCE-4107 Fix tests in org.apache.hadoop.ipc.TestSocketFactory Major mrv2 Devaraj K Devaraj K
YARN-202 Log Aggregation generates a storm of fsync() for namenode Critical . Kihwal Lee Kihwal Lee
YARN-186 Coverage fixing LinuxContainerExecutor Major resourcemanager, scheduler Aleksey Gorshkov Aleksey Gorshkov
YARN-188 Coverage fixing for CapacityScheduler Major capacityscheduler Aleksey Gorshkov Aleksey Gorshkov
YARN-219 NM should aggregate logs when application finishes. Critical nodemanager Robert Joseph Evans Robert Joseph Evans

OTHER:

JIRA Summary Priority Component Reporter Contributor
MAPREDUCE-4596 Split StateMachine state from states seen by MRClientProtocol (for Job, Task, TaskAttempt) Major applicationmaster, mrv2 Siddharth Seth Siddharth Seth
YARN-154 Create Yarn trunk and commit jobs Major . Eli Collins Robert Joseph Evans
MAPREDUCE-4266 remove Ant remnants from MR Major build Alejandro Abdelnur Alejandro Abdelnur
HADOOP-8860 Split MapReduce and YARN sections in documentation navigation Major documentation Tom White Tom White