Hadoop 2.7.5 Release Notes

• YARN-7084. Major bug reported by Jason Lowe and fixed by Jason Lowe
  TestSchedulingMonitor#testRMStarts fails sporadically
  

• YARN-6959. Major bug reported by Yuqi Wang and fixed by Yuqi Wang (capacity scheduler , fairscheduler , scheduler)
  RM may allocate wrong AM Container for new attempt
  

  ResourceManager will now record ResourceRequests from different attempts into different objects.

• YARN-5195. Major bug reported by Karam Singh and fixed by sandflee (resourcemanager)
  RM intermittently crashed with NPE while handling APP_ATTEMPT_REMOVED event when async-scheduling enabled in CapacityScheduler
  

• MAPREDUCE-6975. Major improvement reported by Prabhu Joseph and fixed by Prabhu Joseph (task)
  Logging task counters
  

• MAPREDUCE-6957. Major bug reported by Jooseong Kim and fixed by Jooseong Kim (mrv2)
  shuffle hangs after a node manager connection timeout
  

• MAPREDUCE-6937. Major improvement reported by Zhe Zhang and fixed by Peter Bacsko
  Backport MAPREDUCE-6870 to branch-2 while preserving compatibility
  

• MAPREDUCE-6931. Critical bug reported by Dennis Huo and fixed by Dennis Huo (benchmarks , test)
  Remove TestDFSIO "Total Throughput" calculation
  

• MAPREDUCE-6750. Minor bug reported by Kihwal Lee and fixed by Kihwal Lee (test)
  TestHSAdminServer.testRefreshSuperUserGroups is failing
  

• MAPREDUCE-6165. Minor bug reported by Wei Yan and fixed by Akira Ajisaka
  [JDK8] TestCombineFileInputFormat failed on JDK8
  

• HDFS-12832. Critical bug reported by DENG FEI and fixed by Konstantin Shvachko (namenode)
  INode.getFullPathName may throw ArrayIndexOutOfBoundsException lead to NameNode exit
  

• HDFS-12823. Major improvement reported by Erik Krogen and fixed by Erik Krogen (hdfs , hdfs-client)
  Backport HDFS-9259 "Make SO_SNDBUF size configurable at DFSClient" to branch-2.7
  

• HDFS-12638. Blocker bug reported by Jiandan Yang and fixed by Konstantin Shvachko (hdfs)
  Delete copy-on-truncate block along with the original block, when deleting a file being truncated
  

• HDFS-12596. Major test reported by Xiao Chen and fixed by Xiao Chen (test)
  Add TestFsck#testFsckCorruptWhenOneReplicaIsCorrupt back to branch-2.7
  

• HDFS-12578. Blocker bug reported by Xiao Chen and fixed by Ajay Kumar (test)
  TestDeadDatanode#testNonDFSUsedONDeadNodeReReg failing in branch-2.7
  

• HDFS-12420. Major improvement reported by Ajay Kumar and fixed by Ajay Kumar
  Add an option to disallow 'namenode format -force'
  

• HDFS-12323. Major bug reported by Erik Krogen and fixed by Erik Krogen (namenode , qjm)
  NameNode terminates after full GC thinking QJM unresponsive if full GC is much longer than timeout
  

• HDFS-12157. Critical bug reported by Vinayakumar B and fixed by Vinayakumar B (datanode)
  Do fsyncDirectory(..) outside of FSDataset lock
  

• HDFS-12131. Minor improvement reported by Erik Krogen and fixed by Erik Krogen (hdfs , namenode)
  Add some of the FSNamesystem JMX values as metrics
  

• HDFS-10984. Major task reported by Siddharth Wagle and fixed by Siddharth Wagle (namenode)
  Expose nntop output as metrics
  

• HDFS-10738. Major bug reported by Rakesh R and fixed by Rakesh R (test)
  Fix TestRefreshUserMappings.testRefreshSuperUserGroupsConfiguration test failure
  

• HDFS-10711. Major sub-task reported by Daryn Sharp and fixed by Daryn Sharp (hdfs)
  Optimize FSPermissionChecker group membership check
  

• HDFS-9153. Major test reported by Kai Zheng and fixed by Kai Zheng
  Pretty-format the output for DFSIO
  

• HDFS-9107. Critical bug reported by Daryn Sharp and fixed by Daryn Sharp (namenode)
  Prevent NN's unrecoverable death spiral after full GC
  

• HDFS-9003. Major bug reported by Kihwal Lee and fixed by Kihwal Lee
  ForkJoin thread pool leaks
  

• HDFS-8865. Major improvement reported by Kihwal Lee and fixed by Kihwal Lee
  Improve quota initialization performance
  

• HDFS-8829. Major improvement reported by He Tianyi and fixed by He Tianyi (datanode)
  Make SO_RCVBUF and SO_SNDBUF size configurable for DataTransferProtocol sockets and allow configuring auto-tuning
  

  HDFS-8829 introduces two new configuration settings: dfs.datanode.transfer.socket.send.buffer.size and dfs.datanode.transfer.socket.recv.buffer.size. These settings can be used to control the socket send buffer and receive buffer sizes respectively on the DataNode for client-DataNode and DataNode-DataNode connections. The default values of both settings are 128KB for backwards compatibility. For optimum performance it is recommended to set these values to zero to enable the OS networking stack to auto-tune buffer sizes.

• HDFS-8797. Major bug reported by Jing Zhao and fixed by Jing Zhao (webhdfs)
  WebHdfsFileSystem creates too many connections for pread
  

• HADOOP-14919. Critical bug reported by Aki Tanaka and fixed by Jason Lowe
  BZip2 drops records when reading data in splits
  

• HADOOP-14902. Major bug reported by Jason Lowe and fixed by Hanisha Koneru (fs)
  LoadGenerator#genFile write close timing is incorrectly calculated
  

• HADOOP-14881. Major sub-task reported by Chetna Chaudhari and fixed by Bharat Viswanadham
  LoadGenerator should use Time.monotonicNow() to measure durations
  

• HADOOP-14867. Major bug reported by Bharat Viswanadham and fixed by Bharat Viswanadham
  Update HDFS Federation setup document, for incorrect property name for secondary name node http address
  

• HADOOP-14827. Minor improvement reported by Erik Krogen and fixed by Erik Krogen (common , test)
  Allow StopWatch to accept a Timer parameter for tests
  

• HADOOP-14702. Minor bug reported by Doris Gu and fixed by Doris Gu (documentation)
  Fix formatting issue and regression caused by conversion from APT to Markdown
  

• HADOOP-13442. Major improvement reported by Daryn Sharp and fixed by Daryn Sharp
  Optimize UGI group lookups