Hadoop 2.6.0 Release Notes

• YARN-2853. Major bug reported by Jian He and fixed by Jian He
  Killing app may hang while AM is unregistering
  

• YARN-2846. Blocker bug reported by Junping Du and fixed by Junping Du (nodemanager)
  Incorrect persist exit code for running containers in reacquireContainer() that interrupted by NodeManager restart.
  

• YARN-2843. Major sub-task reported by Sushmitha Sreenivasan and fixed by Wangda Tan (resourcemanager)
  NodeLabels manager should trim all inputs for hosts and labels
  

• YARN-2841. Critical sub-task reported by Jian He and fixed by Jian He (resourcemanager)
  RMProxy should retry EOFException
  

• YARN-2834. Blocker bug reported by Yesha Vora and fixed by Jian He
  Resource manager crashed with Null Pointer Exception
  

• YARN-2830. Blocker bug reported by Jonathan Eagles and fixed by Jonathan Eagles
  Add backwords compatible ContainerId.newInstance constructor for use within Tez Local Mode
  

  I just committed this. Thanks [~jeagles] for the patch and [~ozawa] for the reviews!

• YARN-2827. Critical bug reported by Wangda Tan and fixed by Wangda Tan (client)
  Fix bugs of yarn queue CLI
  

• YARN-2826. Critical bug reported by Sidharta Seethana and fixed by Wangda Tan
  User-Group mappings not updated by RM when a user is removed from a group.
  

• YARN-2825. Critical bug reported by Jian He and fixed by Jian He
  Container leak on NM
  

• YARN-2824. Critical sub-task reported by Wangda Tan and fixed by Wangda Tan (resourcemanager)
  Capacity of labels should be zero by default
  

• YARN-2823. Critical bug reported by Gour Saha and fixed by Jian He (resourcemanager)
  NullPointerException in RM HA enabled 3-node cluster
  

• YARN-2819. Critical bug reported by Gopal V and fixed by Zhijie Shen (timelineserver)
  NPE in ATS Timeline Domains when upgrading from 2.4 to 2.6
  

• YARN-2818. Critical bug reported by Zhijie Shen and fixed by Zhijie Shen (timelineserver)
  Remove the logic to inject entity owner as the primary filter
  

• YARN-2813. Major bug reported by Zhijie Shen and fixed by Zhijie Shen (timelineserver)
  NPE from MemoryTimelineStore.getDomains
  

• YARN-2812. Major test reported by Zhijie Shen and fixed by Zhijie Shen (timelineserver)
  TestApplicationHistoryServer is likely to fail on less powerful machine
  

• YARN-2810. Major test reported by Varun Vasudev and fixed by Varun Vasudev (resourcemanager)
  TestRMProxyUsersConf fails on Windows VMs
  

• YARN-2805. Blocker bug reported by Arpit Gupta and fixed by Wangda Tan (resourcemanager)
  RM2 in HA setup tries to login using the RM1's kerberos principal
  

• YARN-2804. Critical bug reported by Zhijie Shen and fixed by Zhijie Shen
  Timeline server .out log have JAXB binding exceptions and warnings.
  

• YARN-2803. Critical bug reported by Chris Nauroth and fixed by Craig Welch (nodemanager)
  MR distributed cache not working correctly on Windows after NodeManager privileged account changes.
  

• YARN-2798. Blocker bug reported by Arpit Gupta and fixed by Zhijie Shen (timelineserver)
  YarnClient doesn't need to translate Kerberos name of timeline DT renewer
  

• YARN-2795. Major sub-task reported by Phil D'Amore and fixed by Wangda Tan (resourcemanager)
  Resource Manager fails startup with HDFS label storage and secure cluster
  

• YARN-2794. Major bug reported by Jian He and fixed by Jian He
  Fix log msgs about distributing system-credentials
  

• YARN-2790. Critical bug reported by Tassapol Athiapinya and fixed by Jian He (nodemanager)
  NM can't aggregate logs past HDFS delegation token expiry.
  

• YARN-2789. Critical task reported by Siddharth Seth and fixed by Wangda Tan
  Re-instate the NodeReport.newInstance API modified in YARN-2698
  

• YARN-2788. Blocker bug reported by Gopal V and fixed by Xuan Gong (log-aggregation)
  yarn logs -applicationId on 2.6.0 should support logs written by 2.4.0
  

• YARN-2785. Major test reported by Varun Vasudev and fixed by Varun Vasudev
  TestContainerResourceUsage fails intermittently
  

• YARN-2779. Critical bug reported by Zhijie Shen and fixed by Zhijie Shen (resourcemanager , timelineserver)
  SystemMetricsPublisher can use Kerberos directly instead of timeline DT
  

• YARN-2778. Major sub-task reported by Wangda Tan and fixed by Wangda Tan (client)
  YARN node CLI should display labels on returned node reports
  

• YARN-2770. Critical sub-task reported by Zhijie Shen and fixed by Zhijie Shen (timelineserver)
  Timeline delegation tokens need to be automatically renewed by the RM
  

• YARN-2769. Major bug reported by Varun Vasudev and fixed by Varun Vasudev (applications/distributed-shell)
  Timeline server domain not set correctly when using shell_command on Windows
  

• YARN-2767. Major test reported by Varun Vasudev and fixed by Varun Vasudev (resourcemanager)
  RM web services - add test case to ensure the http static user cannot kill or submit apps in secure mode
  

• YARN-2760. Trivial bug reported by Harsh J and fixed by Harsh J (documentation)
  Completely remove word 'experimental' from FairScheduler docs
  

• YARN-2758. Major test reported by Zhijie Shen and fixed by Zhijie Shen (timelineserver)
  Update TestApplicationHistoryClientService to use the new generic history store
  

• YARN-2755. Critical bug reported by Siqi Li and fixed by Siqi Li
  NM fails to clean up usercache_DEL_<timestamp> dirs after YARN-661
  

• YARN-2753. Major sub-task reported by zhihai xu and fixed by zhihai xu
  Fix potential issues and code clean up for *NodeLabelsManager
  

• YARN-2752. Critical bug reported by Xuan Gong and fixed by Xuan Gong
  ContainerExecutor always append "nice -n" in command on branch-2
  

• YARN-2747. Major test reported by Xuan Gong and fixed by Xuan Gong
  TestAggregatedLogFormat fails in trunk
  

• YARN-2744. Critical sub-task reported by Sumit Mohanty and fixed by Wangda Tan (capacityscheduler)
  Under some scenario, it is possible to end up with capacity scheduler configuration that uses labels that no longer exist
  

• YARN-2743. Blocker bug reported by Arpit Gupta and fixed by Jian He (resourcemanager)
  Yarn jobs via oozie fail with failed to renew token (secure) or digest mismatch (unsecure) errors when RM is being killed
  

• YARN-2741. Major bug reported by Craig Welch and fixed by Craig Welch (nodemanager)
  Windows: Node manager cannot serve up log files via the web user interface when yarn.nodemanager.log-dirs to any drive letter other than C: (or, the drive that nodemanager is running on)
  

• YARN-2734. Major bug reported by Sumit Mohanty and fixed by Xuan Gong (log-aggregation)
  If a sub-folder is encountered by log aggregator it results in invalid aggregated file
  

• YARN-2732. Major bug reported by Jian He and fixed by Jian He
  Fix syntax error in SecureContainer.apt.vm
  

• YARN-2730. Critical bug reported by Siqi Li and fixed by Siqi Li
  DefaultContainerExecutor runs only one localizer at a time
  

• YARN-2726. Minor sub-task reported by Phil D'Amore and fixed by Wangda Tan (capacityscheduler)
  CapacityScheduler should explicitly log when an accessible label has no capacity
  

• YARN-2724. Major bug reported by Sumit Mohanty and fixed by Xuan Gong (log-aggregation)
  If an unreadable file is encountered during log aggregation then aggregated file in HDFS badly formed
  

• YARN-2723. Major sub-task reported by Phil D'Amore and fixed by Naganarasimha G R (client)
  rmadmin -replaceLabelsOnNode does not correctly parse port
  

• YARN-2721. Blocker bug reported by Jian He and fixed by Jian He
  Race condition: ZKRMStateStore retry logic may throw NodeExist exception
  

• YARN-2720. Major bug reported by Craig Welch and fixed by Craig Welch (nodemanager)
  Windows: Wildcard classpath variables not expanded against resources contained in archives
  

• YARN-2717. Major sub-task reported by Xuan Gong and fixed by Xuan Gong (log-aggregation)
  containerLogNotFound log shows multiple time for the same container
  

• YARN-2715. Blocker bug reported by Zhijie Shen and fixed by Zhijie Shen (resourcemanager)
  Proxy user is problem for RPC interface if yarn.resourcemanager.webapp.proxyuser is not set.
  

• YARN-2711. Major test reported by Varun Vasudev and fixed by Varun Vasudev
  TestDefaultContainerExecutor#testContainerLaunchError fails on Windows
  

• YARN-2709. Major sub-task reported by Li Lu and fixed by Li Lu
  Add retry for timeline client getDelegationToken method
  

• YARN-2707. Minor bug reported by Ted Yu and fixed by Gera Shegalov
  Potential null dereference in FSDownload
  

• YARN-2705. Critical sub-task reported by Wangda Tan and fixed by Wangda Tan (api , client , resourcemanager)
  Changes of RM node label manager default configuration
  

• YARN-2704. Critical sub-task reported by Jian He and fixed by Jian He
  Localization and log-aggregation will fail if hdfs delegation token expired after token-max-life-time
  

• YARN-2703. Major sub-task reported by Xuan Gong and fixed by Xuan Gong (nodemanager , resourcemanager)
  Add logUploadedTime into LogValue for better display
  

• YARN-2701. Blocker bug reported by Xuan Gong and fixed by Xuan Gong
  Potential race condition in startLocalizer when using LinuxContainerExecutor
  

• YARN-2700. Major sub-task reported by Steve Loughran and fixed by Steve Loughran (api , resourcemanager)
  TestSecureRMRegistryOperations failing on windows: auth problems
  

• YARN-2699. Blocker sub-task reported by Wangda Tan and fixed by Wangda Tan (client)
  Fix test timeout in TestResourceTrackerOnHA#testResourceTrackerOnHA
  

• YARN-2698. Critical sub-task reported by Wangda Tan and fixed by Wangda Tan (api , client , resourcemanager)
  Move getClusterNodeLabels and getNodeToLabels to YarnClient instead of AdminService
  

• YARN-2692. Major sub-task reported by Steve Loughran and fixed by Steve Loughran
  ktutil test hanging on some machines/ktutil versions
  

• YARN-2689. Major sub-task reported by Steve Loughran and fixed by Steve Loughran (api , resourcemanager)
  TestSecureRMRegistryOperations failing on windows: secure ZK won't start
  

• YARN-2685. Major sub-task reported by Wangda Tan and fixed by Wangda Tan (api , client , resourcemanager)
  Resource on each label not correct when multiple NMs in a same host and some has label some not
  

• YARN-2682. Minor bug reported by zhihai xu and fixed by zhihai xu (nodemanager)
  WindowsSecureContainerExecutor should not depend on DefaultContainerExecutor#getFirstApplicationDir.
  

• YARN-2678. Major sub-task reported by Gour Saha and fixed by Steve Loughran (api , resourcemanager)
  Improved Yarn Registry service record structure
  

• YARN-2677. Major sub-task reported by Steve Loughran and fixed by Steve Loughran (api , resourcemanager)
  registry punycoding of usernames doesn't fix all usernames to be DNS-valid
  

• YARN-2676. Major sub-task reported by Zhijie Shen and fixed by Zhijie Shen (timelineserver)
  Timeline authentication filter should add support for proxy user
  

• YARN-2673. Major sub-task reported by Li Lu and fixed by Li Lu
  Add retry for timeline client put APIs
  

• YARN-2671. Blocker bug reported by Zhijie Shen and fixed by Wangda Tan (resourcemanager)
  ApplicationSubmissionContext change breaks the existing app submission
  

• YARN-2668. Major sub-task reported by Steve Loughran and fixed by Steve Loughran (client)
  yarn-registry JAR won't link against ZK 3.4.5
  

• YARN-2667. Minor bug reported by Yi Liu and fixed by Yi Liu
  Fix the release audit warning caused by hadoop-yarn-registry
  

• YARN-2662. Minor bug reported by Chris Nauroth and fixed by Chris Nauroth (test)
  TestCgroupsLCEResourcesHandler leaks file descriptors.
  

• YARN-2656. Major bug reported by Varun Vasudev and fixed by Zhijie Shen (resourcemanager)
  RM web services authentication filter should add support for proxy user
  

• YARN-2652. Major sub-task reported by Steve Loughran and fixed by Steve Loughran (api)
  add hadoop-yarn-registry package under hadoop-yarn
  

• YARN-2651. Major sub-task reported by Xuan Gong and fixed by Xuan Gong (nodemanager , resourcemanager)
  Spin off the LogRollingInterval from LogAggregationContext
  

• YARN-2649. Major bug reported by Ming Ma and fixed by Ming Ma
  Flaky test TestAMRMRPCNodeUpdates
  

• YARN-2647. Major sub-task reported by Wangda Tan and fixed by Sunil G (client)
  Add yarn queue CLI to get queue infos
  

• YARN-2644. Major sub-task reported by Craig Welch and fixed by Craig Welch
  Recalculate headroom more frequently to keep it accurate
  

• YARN-2635. Major bug reported by Wei Yan and fixed by Wei Yan
  TestRM, TestRMRestart, TestClientToAMTokens should run with both CS and FS
  

• YARN-2632. Blocker sub-task reported by Junping Du and fixed by Junping Du (nodemanager)
  Document NM Restart feature
  

• YARN-2630. Major bug reported by Jian He and fixed by Jian He
  TestDistributedShell#testDSRestartWithPreviousRunningContainers fails
  

• YARN-2629. Major sub-task reported by Zhijie Shen and fixed by Zhijie Shen (timelineserver)
  Make distributed shell use the domain-based timeline ACLs
  

• YARN-2628. Major bug reported by Varun Vasudev and fixed by Varun Vasudev (capacityscheduler)
  Capacity scheduler with DominantResourceCalculator carries out reservation even though slots are free
  

• YARN-2627. Major improvement reported by Xuan Gong and fixed by Xuan Gong
  Add logs when attemptFailuresValidityInterval is enabled
  

• YARN-2624. Blocker bug reported by Anubhav Dhoot and fixed by Anubhav Dhoot (nodemanager)
  Resource Localization fails on a cluster due to existing cache directories
  

• YARN-2621. Major sub-task reported by Zhijie Shen and fixed by Zhijie Shen (timelineserver)
  Simplify the output when the user doesn't have the access for getDomain(s)
  

• YARN-2617. Major bug reported by Jun Gong and fixed by Jun Gong (nodemanager)
  NM does not need to send finished container whose APP is not running to RM
  

• YARN-2615. Blocker sub-task reported by Junping Du and fixed by Junping Du
  ClientToAMTokenIdentifier and DelegationTokenIdentifier should allow extended fields
  

• YARN-2611. Major sub-task reported by Subru Krishnan and fixed by Subru Krishnan (capacityscheduler , resourcemanager , scheduler)
  Fix jenkins findbugs warning and test case failures for trunk merge patch
  

• YARN-2610. Major bug reported by Ray Chiang and fixed by Ray Chiang
  Hamlet should close table tags
  

• YARN-2608. Major bug reported by Wei Yan and fixed by Wei Yan
  FairScheduler: Potential deadlocks in loading alloc files and clock access
  

• YARN-2607. Major test reported by Ted Yu and fixed by Wangda Tan
  TestDistributedShell fails in trunk
  

• YARN-2606. Major bug reported by Mit Desai and fixed by Mit Desai (timelineserver)
  Application History Server tries to access hdfs before doing secure login
  

• YARN-2602. Major bug reported by Karam Singh and fixed by Zhijie Shen (timelineserver)
  Generic History Service of TimelineServer sometimes not able to handle NPE
  

• YARN-2596. Major test reported by Junping Du and fixed by Karthik Kambatla
  TestWorkPreservingRMRestart fails with FairScheduler
  

• YARN-2594. Blocker bug reported by Karam Singh and fixed by Wangda Tan (resourcemanager)
  Potential deadlock in RM when querying ApplicationResourceUsageReport
  

• YARN-2591. Major sub-task reported by Zhijie Shen and fixed by Zhijie Shen (timelineserver)
  AHSWebServices should return FORBIDDEN(403) if the request user doesn't have access to the history data
  

• YARN-2588. Major bug reported by Rohith and fixed by Rohith (resourcemanager)
  Standby RM does not transitionToActive if previous transitionToActive is failed with ZK exception.
  

• YARN-2584. Major test reported by Zhijie Shen and fixed by Jian He
  TestContainerManagerSecurity fails on trunk
  

• YARN-2583. Major sub-task reported by Xuan Gong and fixed by Xuan Gong (nodemanager , resourcemanager)
  Modify the LogDeletionService to support Log aggregation for LRS
  

• YARN-2582. Major sub-task reported by Xuan Gong and fixed by Xuan Gong (nodemanager , resourcemanager)
  Log related CLI and Web UI changes for Aggregated Logs in LRS
  

• YARN-2581. Major sub-task reported by Xuan Gong and fixed by Xuan Gong (nodemanager , resourcemanager)
  NMs need to find a way to get LogAggregationContext
  

• YARN-2579. Blocker bug reported by Rohith and fixed by Rohith (resourcemanager)
  Deadlock when EmbeddedElectorService and FatalEventDispatcher try to transition RM to StandBy at the same time
  

• YARN-2577. Trivial improvement reported by Miklos Christine and fixed by Miklos Christine (documentation , fairscheduler)
  Clarify ACL delimiter and how to configure ACL groups only
  

• YARN-2576. Major sub-task reported by Subru Krishnan and fixed by Subru Krishnan (capacityscheduler , resourcemanager , scheduler)
  Prepare yarn-1051 branch for merging with trunk
  

• YARN-2569. Major sub-task reported by Xuan Gong and fixed by Xuan Gong (nodemanager , resourcemanager)
  API changes for handling logs of long-running services
  

• YARN-2568. Major bug reported by Jian He and fixed by Jian He
  TestAMRMClientOnRMRestart test fails
  

• YARN-2566. Critical sub-task reported by zhihai xu and fixed by zhihai xu (nodemanager)
  DefaultContainerExecutor should pick a working directory randomly
  

• YARN-2565. Major bug reported by Karam Singh and fixed by Zhijie Shen (resourcemanager , timelineserver)
  RM shouldn't use the old RMApplicationHistoryWriter unless explicitly setting FileSystemApplicationHistoryStore
  

• YARN-2563. Blocker bug reported by Arpit Gupta and fixed by Zhijie Shen (timelineserver)
  On secure clusters call to timeline server fails with authentication errors when running a job via oozie
  

• YARN-2562. Critical bug reported by Vinod Kumar Vavilapalli and fixed by Tsuyoshi OZAWA
  ContainerId@toString() is unreadable for epoch >0 after YARN-2182
  

• YARN-2561. Blocker sub-task reported by Tassapol Athiapinya and fixed by Junping Du
  MR job client cannot reconnect to AM after NM restart.
  

• YARN-2559. Major bug reported by Karam Singh and fixed by Zhijie Shen (resourcemanager , timelineserver)
  ResourceManager sometime become un-responsive due to NPE in SystemMetricsPublisher
  

• YARN-2558. Blocker sub-task reported by Tsuyoshi OZAWA and fixed by Tsuyoshi OZAWA
  Updating ContainerTokenIdentifier#read/write to use ContainerId#getContainerId
  

• YARN-2557. Major bug reported by Xuan Gong and fixed by Xuan Gong (applications/distributed-shell)
  Add a parameter "attempt_Failures_Validity_Interval" in DistributedShell
  

• YARN-2549. Minor test reported by Chris Nauroth and fixed by Chris Nauroth (nodemanager , test)
  TestContainerLaunch fails due to classpath problem with hamcrest classes.
  

• YARN-2547. Major sub-task reported by Jonathan Eagles and fixed by Mit Desai (timelineserver)
  Cross Origin Filter throws UnsupportedOperationException upon destroy
  

• YARN-2546. Major bug reported by Doug Haigh and fixed by Varun Vasudev (api)
  REST API for application creation/submission is using strings for numeric & boolean values
  

• YARN-2544. Major sub-task reported by Wangda Tan and fixed by Wangda Tan (api , client , resourcemanager)
  Common server side PB changes (not include user API PB changes)
  

• YARN-2542. Major sub-task reported by Zhijie Shen and fixed by Zhijie Shen
  "yarn application -status <appId>" throws NPE when retrieving the app from the timelineserver
  

• YARN-2541. Major bug reported by Jian He and fixed by Jian He
  Fix ResourceManagerRest.apt.vm syntax error
  

• YARN-2540. Major bug reported by Ashwin Shankar and fixed by Ashwin Shankar (scheduler)
  FairScheduler: Queue filters not working on scheduler page in RM UI
  

• YARN-2539. Minor improvement reported by Wei Yan and fixed by Wei Yan
  FairScheduler: Set the default value for maxAMShare to 0.5
  

• YARN-2538. Major sub-task reported by Xuan Gong and fixed by Xuan Gong (resourcemanager)
  Add logs when RM send new AMRMToken to ApplicationMaster
  

• YARN-2534. Major bug reported by zhihai xu and fixed by zhihai xu (scheduler)
  FairScheduler: Potential integer overflow calculating totalMaxShare
  

• YARN-2531. Major improvement reported by Varun Vasudev and fixed by Varun Vasudev
  CGroups - Admins should be allowed to enforce strict cpu limits
  

• YARN-2529. Major sub-task reported by Zhijie Shen and fixed by Zhijie Shen (timelineserver)
  Generic history service RPC interface doesn't work when service authorization is enabled
  

• YARN-2528. Major sub-task reported by Jonathan Eagles and fixed by Jonathan Eagles (timelineserver)
  Cross Origin Filter Http response split vulnerability protection rejects valid origins
  

• YARN-2527. Major bug reported by Benoy Antony and fixed by Benoy Antony (resourcemanager)
  NPE in ApplicationACLsManager
  

• YARN-2526. Critical bug reported by Wei Yan and fixed by Wei Yan (scheduler-load-simulator)
  SLS can deadlock when all the threads are taken by AMSimulators
  

• YARN-2523. Major bug reported by Nishan Shetty and fixed by Rohith (resourcemanager , webapp)
  ResourceManager UI showing negative value for "Decommissioned Nodes" field
  

• YARN-2519. Major test reported by Xiaoyu Yao and fixed by Xiaoyu Yao (webapp)
  Credential Provider related unit tests failed on Windows
  

• YARN-2515. Major sub-task reported by Tsuyoshi OZAWA and fixed by Tsuyoshi OZAWA (resourcemanager)
  Update ConverterUtils#toContainerId to parse epoch
  

• YARN-2512. Major sub-task reported by Jonathan Eagles and fixed by Jonathan Eagles (timelineserver)
  Allow for origin pattern matching in cross origin filter
  

• YARN-2511. Major sub-task reported by Jonathan Eagles and fixed by Jonathan Eagles (timelineserver)
  Allow All Origins by default when Cross Origin Filter is enabled
  

• YARN-2509. Major sub-task reported by Jonathan Eagles and fixed by Mit Desai (timelineserver)
  Enable Cross Origin Filter for timeline server only and not all Yarn servers
  

• YARN-2508. Major sub-task reported by Jonathan Eagles and fixed by Mit Desai (timelineserver)
  Cross Origin configuration parameters prefix are not honored
  

• YARN-2507. Major sub-task reported by Jonathan Eagles and fixed by Jonathan Eagles (documentation , timelineserver)
  Document Cross Origin Filter Configuration for ATS
  

• YARN-2505. Major sub-task reported by Wangda Tan and fixed by Craig Welch (resourcemanager)
  Support get/add/remove/change labels in RM REST API
  

• YARN-2504. Critical sub-task reported by Wangda Tan and fixed by Wangda Tan (resourcemanager)
  Support get/add/remove/change labels in RM admin CLI
  

• YARN-2503. Major sub-task reported by Wangda Tan and fixed by Wangda Tan (resourcemanager)
  Changes in RM Web UI to better show labels to end users
  

• YARN-2502. Major sub-task reported by Wangda Tan and fixed by Wangda Tan (resourcemanager)
  Changes in distributed shell to support specify labels
  

• YARN-2501. Major sub-task reported by Wangda Tan and fixed by Wangda Tan (resourcemanager)
  Changes in AMRMClient to support labels
  

• YARN-2500. Major sub-task reported by Wangda Tan and fixed by Wangda Tan (resourcemanager)
  Miscellaneous changes in ResourceManager to support labels
  

• YARN-2496. Major sub-task reported by Wangda Tan and fixed by Wangda Tan (resourcemanager)
  Changes for capacity scheduler to support allocate resource respect labels
  

• YARN-2494. Major sub-task reported by Wangda Tan and fixed by Wangda Tan (resourcemanager)
  Node label manager API and storage implementations
  

• YARN-2493. Major sub-task reported by Wangda Tan and fixed by Wangda Tan (api)
  API changes for users
  

• YARN-2484. Trivial bug reported by Tsuyoshi OZAWA and fixed by Tsuyoshi OZAWA
  FileSystemRMStateStore#readFile/writeFile should close FSData(In|Out)putStream in final block
  

• YARN-2475. Major sub-task reported by Carlo Curino and fixed by Carlo Curino (resourcemanager)
  ReservationSystem: replan upon capacity reduction
  

• YARN-2468. Major sub-task reported by Xuan Gong and fixed by Xuan Gong (log-aggregation , nodemanager , resourcemanager)
  Log handling for LRS
  

• YARN-2462. Major bug reported by Jason Lowe and fixed by Eric Payne
  TestNodeManagerResync#testBlockNewContainerRequestsOnStartAndResync should have a test timeout
  

• YARN-2460. Minor bug reported by Ray Chiang and fixed by Ray Chiang
  Remove obsolete entries from yarn-default.xml
  

• YARN-2459. Major bug reported by Mayank Bansal and fixed by Mayank Bansal (resourcemanager)
  RM crashes if App gets rejected for any reason and HA is enabled
  

• YARN-2456. Major sub-task reported by Jian He and fixed by Jian He (resourcemanager)
  Possible livelock in CapacityScheduler when RM is recovering apps
  

• YARN-2453. Major bug reported by zhihai xu and fixed by zhihai xu
  TestProportionalCapacityPreemptionPolicy fails with FairScheduler
  

• YARN-2452. Major bug reported by zhihai xu and fixed by zhihai xu
  TestRMApplicationHistoryWriter fails with FairScheduler
  

• YARN-2450. Trivial bug reported by Ray Chiang and fixed by Ray Chiang
  Fix typos in log messages
  

• YARN-2449. Critical bug reported by Karam Singh and fixed by Varun Vasudev (timelineserver)
  Timelineserver returns invalid Delegation token in secure kerberos enabled cluster when hadoop.http.filter.initializers are not set
  

• YARN-2448. Major improvement reported by Varun Vasudev and fixed by Varun Vasudev
  RM should expose the resource types considered during scheduling when AMs register
  

• YARN-2447. Major bug reported by Varun Vasudev and fixed by Varun Vasudev
  RM web services app submission doesn't pass secrets correctly
  

• YARN-2446. Major sub-task reported by Zhijie Shen and fixed by Zhijie Shen (timelineserver)
  Using TimelineNamespace to shield the entities of a user
  

• YARN-2440. Major bug reported by Varun Vasudev and fixed by Varun Vasudev
  Cgroups should allow YARN containers to be limited to allocated cores
  

• YARN-2434. Major sub-task reported by Jian He and fixed by Jian He
  RM should not recover containers from previously failed attempt when AM restart is not enabled
  

• YARN-2431. Major sub-task reported by Jason Lowe and fixed by Jason Lowe (nodemanager)
  NM restart: cgroup is not removed for reacquired containers
  

• YARN-2424. Blocker bug reported by Allen Wittenauer and fixed by Allen Wittenauer (nodemanager)
  LCE should support non-cgroups, non-secure mode
  

• YARN-2411. Major improvement reported by Ram Venkatesh and fixed by Ram Venkatesh (capacityscheduler)
  [Capacity Scheduler] support simple user and group mappings to queues
  

• YARN-2409. Critical bug reported by Nishan Shetty and fixed by Rohith (resourcemanager)
  Active to StandBy transition does not stop rmDispatcher that causes 1 AsyncDispatcher thread leak.
  

• YARN-2406. Major sub-task reported by Jian He and fixed by Tsuyoshi OZAWA
  Move RM recovery related proto to yarn_server_resourcemanager_recovery.proto
  

• YARN-2405. Major bug reported by Maysam Yabandeh and fixed by Tsuyoshi OZAWA
  NPE in FairSchedulerAppsBlock
  

• YARN-2400. Major bug reported by Jian He and fixed by Jian He
  TestAMRestart fails intermittently
  

• YARN-2399. Major improvement reported by Karthik Kambatla and fixed by Karthik Kambatla (fairscheduler)
  FairScheduler: Merge AppSchedulable and FSSchedulerApp into FSAppAttempt
  

• YARN-2397. Critical bug reported by Varun Vasudev and fixed by Varun Vasudev
  RM and TS web interfaces sometimes return request is a replay error in secure mode
  

• YARN-2395. Major new feature reported by Ashwin Shankar and fixed by Wei Yan (fairscheduler)
  FairScheduler: Preemption timeout should be configurable per queue
  

• YARN-2394. Major new feature reported by Ashwin Shankar and fixed by Wei Yan (fairscheduler)
  FairScheduler: Configure fairSharePreemptionThreshold per queue
  

• YARN-2393. Major new feature reported by Ashwin Shankar and fixed by Wei Yan (fairscheduler)
  FairScheduler: Add the notion of steady fair share
  

• YARN-2389. Major sub-task reported by Subru Krishnan and fixed by Subru Krishnan (capacityscheduler , fairscheduler)
  Adding support for drainig a queue, ie killing all apps in the queue
  

• YARN-2388. Major test reported by Zhijie Shen and fixed by Zhijie Shen
  TestTimelineWebServices fails on trunk after HADOOP-10791
  

• YARN-2387. Blocker bug reported by Mit Desai and fixed by Mit Desai
  Resource Manager crashes with NPE due to lack of synchronization
  

• YARN-2378. Major sub-task reported by Subru Krishnan and fixed by Subru Krishnan (capacityscheduler)
  Adding support for moving apps between queues in Capacity Scheduler
  

• YARN-2377. Major improvement reported by Gera Shegalov and fixed by Gera Shegalov (nodemanager)
  Localization exception stack traces are not passed as diagnostic info
  

• YARN-2374. Major bug reported by Varun Vasudev and fixed by Varun Vasudev
  YARN trunk build failing TestDistributedShell.testDSShell
  

• YARN-2373. Major bug reported by Larry McCay and fixed by Larry McCay
  WebAppUtils Should Use configuration.getPassword for Accessing SSL Passwords
  

• YARN-2372. Minor improvement reported by Fengdong Yu and fixed by Fengdong Yu (documentation)
  There are Chinese Characters in the FairScheduler's document
  

• YARN-2370. Trivial bug reported by Wenwu Peng and fixed by Wenwu Peng (resourcemanager)
  Fix comment in o.a.h.y.server.resourcemanager.schedulerAppSchedulingInfo
  

• YARN-2363. Major bug reported by Jason Lowe and fixed by Jason Lowe (resourcemanager)
  Submitted applications occasionally lack a tracking URL
  

• YARN-2361. Trivial improvement reported by zhihai xu and fixed by zhihai xu (resourcemanager)
  RMAppAttempt state machine entries for KILLED state has duplicate event entries
  

• YARN-2360. Major new feature reported by Ashwin Shankar and fixed by Ashwin Shankar (fairscheduler)
  Fair Scheduler: Display dynamic fair share for queues on the scheduler page
  

• YARN-2359. Critical bug reported by zhihai xu and fixed by zhihai xu (resourcemanager)
  Application hangs when it fails to launch AM container
  

• YARN-2354. Major sub-task reported by Jian He and fixed by Li Lu
  DistributedShell may allocate more containers than client specified after it restarts
  

• YARN-2352. Major improvement reported by Karthik Kambatla and fixed by Karthik Kambatla (scheduler)
  FairScheduler: Collect metrics on duration of critical methods that affect performance
  

• YARN-2347. Major sub-task reported by Junping Du and fixed by Junping Du
  Consolidate RMStateVersion and NMDBSchemaVersion into StateVersion in yarn-server-common
  

• YARN-2343. Trivial improvement reported by Li Lu and fixed by Li Lu
  Improve error message on token expire exception
  

• YARN-2337. Trivial improvement reported by zhihai xu and fixed by zhihai xu (resourcemanager)
  ResourceManager sets ClientRMService in RMContext multiple times
  

• YARN-2323. Minor improvement reported by Hong Zhiguo and fixed by Hong Zhiguo (fairscheduler)
  FairShareComparator creates too many Resource objects
  

• YARN-2321. Major bug reported by Leitao Guo and fixed by Leitao Guo (nodemanager)
  NodeManager web UI can incorrectly report Pmem enforcement
  

• YARN-2317. Major sub-task reported by Li Lu and fixed by Li Lu (documentation)
  Update documentation about how to write YARN applications
  

• YARN-2314. Critical bug reported by Jason Lowe and fixed by Jason Lowe (client)
  ContainerManagementProtocolProxy can create thousands of threads for a large cluster
  

• YARN-2313. Major bug reported by Tsuyoshi OZAWA and fixed by Tsuyoshi OZAWA (fairscheduler)
  Livelock can occur in FairScheduler when there are lots of running apps
  

• YARN-2312. Major sub-task reported by Tsuyoshi OZAWA and fixed by Tsuyoshi OZAWA (resourcemanager)
  Marking ContainerId#getId as deprecated
  

• YARN-2308. Critical bug reported by Wangda Tan and fixed by chang li (resourcemanager , scheduler)
  NPE happened when RM restart after CapacityScheduler queue configuration changed
  

• YARN-2302. Major sub-task reported by Zhijie Shen and fixed by Zhijie Shen (timelineserver)
  Refactor TimelineWebServices
  

• YARN-2298. Major sub-task reported by Zhijie Shen and fixed by Zhijie Shen (client)
  Move TimelineClient to yarn-common project
  

• YARN-2295. Major sub-task reported by Li Lu and fixed by Li Lu
  Refactor YARN distributed shell with existing public stable API
  

• YARN-2288. Major sub-task reported by Junping Du and fixed by Junping Du (timelineserver)
  Data persistent in timelinestore should be versioned
  

• YARN-2279. Major sub-task reported by Zhijie Shen and fixed by Zhijie Shen (timelineserver)
  Add UTs to cover timeline server authentication
  

• YARN-2277. Major sub-task reported by Jonathan Eagles and fixed by Jonathan Eagles (timelineserver)
  Add Cross-Origin support to the ATS REST API
  

• YARN-2274. Trivial improvement reported by Karthik Kambatla and fixed by Karthik Kambatla (fairscheduler)
  FairScheduler: Add debug information about cluster capacity, availability and reservations
  

• YARN-2273. Major bug reported by Andy Skelton and fixed by Wei Yan (fairscheduler , resourcemanager)
  NPE in ContinuousScheduling thread when we lose a node
  

• YARN-2269. Major bug reported by Yesha Vora and fixed by Craig Welch
  External links need to be removed from YARN UI
  

• YARN-2264. Major bug reported by Siddharth Seth and fixed by Li Lu
  Race in DrainDispatcher can cause random test failures
  

• YARN-2260. Major sub-task reported by Jian He and fixed by Jian He (resourcemanager)
  Add containers to launchedContainers list in RMNode on container recovery
  

• YARN-2252. Major bug reported by Ratandeep Ratti and fixed by (scheduler)
  Intermittent failure of TestFairScheduler.testContinuousScheduling
  

• YARN-2251. Major bug reported by Zhijie Shen and fixed by Zhijie Shen
  Avoid negative elapsed time in JHS/MRAM web UI and services
  

• YARN-2249. Major sub-task reported by Jian He and fixed by Jian He (resourcemanager)
  AM release request may be lost on RM restart
  

• YARN-2248. Major sub-task reported by Janos Matyas and fixed by Janos Matyas (capacityscheduler)
  Capacity Scheduler changes for moving apps between queues
  

• YARN-2244. Critical bug reported by Anubhav Dhoot and fixed by Anubhav Dhoot (fairscheduler)
  FairScheduler missing handling of containers for unknown application attempts
  

• YARN-2242. Major sub-task reported by Li Lu and fixed by Li Lu
  Improve exception information on AM launch crashes
  

• YARN-2237. Major sub-task reported by Xuan Gong and fixed by Xuan Gong (resourcemanager)
  MRAppMaster changes for AMRMToken roll-up
  

• YARN-2229. Major sub-task reported by Tsuyoshi OZAWA and fixed by Tsuyoshi OZAWA (resourcemanager)
  ContainerId can overflow with RM restart
  

• YARN-2228. Major sub-task reported by Zhijie Shen and fixed by Zhijie Shen
  TimelineServer should load pseudo authentication filter when authentication = simple
  

• YARN-2219. Major bug reported by Ashwin Shankar and fixed by Jian He (resourcemanager)
  AMs and NMs can get exceptions after recovery but before scheduler knowns apps and app-attempts
  

• YARN-2214. Major improvement reported by Ashwin Shankar and fixed by Ashwin Shankar (scheduler)
  FairScheduler: preemptContainerPreCheck() in FSParentQueue delays convergence towards fairness
  

• YARN-2212. Major sub-task reported by Xuan Gong and fixed by Xuan Gong (resourcemanager)
  ApplicationMaster needs to find a way to update the AMRMToken periodically
  

• YARN-2211. Major sub-task reported by Xuan Gong and fixed by Xuan Gong (resourcemanager)
  RMStateStore needs to save AMRMToken master key for recovery when RM restart/failover happens
  

• YARN-2209. Major improvement reported by Jian He and fixed by Jian He
  Replace AM resync/shutdown command with corresponding exceptions
  

• YARN-2208. Major sub-task reported by Xuan Gong and fixed by Xuan Gong (resourcemanager)
  AMRMTokenManager need to have a way to roll over AMRMToken
  

• YARN-2207. Major task reported by Xuan Gong and fixed by Xuan Gong (resourcemanager)
  Add ability to roll over AMRMToken
  

• YARN-2198. Major improvement reported by Remus Rusanu and fixed by Remus Rusanu
  Remove the need to run NodeManager as privileged account for Windows Secure Container Executor
  

• YARN-2197. Minor improvement reported by Akira AJISAKA and fixed by Akira AJISAKA (documentation)
  Add a link to YARN CHANGES.txt in the left side of doc
  

• YARN-2182. Major sub-task reported by Tsuyoshi OZAWA and fixed by Tsuyoshi OZAWA
  Update ContainerId#toString() to avoid conflicts before and after RM restart
  

• YARN-2181. Major bug reported by Wangda Tan and fixed by Wangda Tan (resourcemanager , webapp)
  Add preemption info to RM Web UI and add logs when preemption occurs
  

• YARN-2174. Major sub-task reported by Zhijie Shen and fixed by Zhijie Shen
  Enabling HTTPs for the writer REST API of TimelineServer
  

• YARN-2161. Major bug reported by Binglin Chang and fixed by Binglin Chang
  Fix build on macosx: YARN parts
  

• YARN-2158. Minor test reported by Ted Yu and fixed by Varun Vasudev
  TestRMWebServicesAppsModification sometimes fails in trunk
  

• YARN-2153. Major sub-task reported by Jian He and fixed by Jian He (resourcemanager)
  Ensure distributed shell work with RM work-preserving recovery
  

• YARN-2147. Minor bug reported by Jason Lowe and fixed by Chen He (resourcemanager)
  client lacks delegation token exception details when application submit fails
  

• YARN-2138. Major bug reported by Jian He and fixed by Varun Saxena
  Cleanup notifyDone* methods in RMStateStore
  

• YARN-2131. Major new feature reported by Karthik Kambatla and fixed by Robert Kanter (resourcemanager)
  Add a way to format the RMStateStore
  

• YARN-2102. Major sub-task reported by Zhijie Shen and fixed by Zhijie Shen
  More generalized timeline ACLs
  

• YARN-2088. Major bug reported by Binglin Chang and fixed by Binglin Chang
  Fix code bug in GetApplicationsRequestPBImpl#mergeLocalToBuilder
  

• YARN-2080. Major sub-task reported by Subru Krishnan and fixed by Subru Krishnan (resourcemanager)
  Admission Control: Integrate Reservation subsystem with ResourceManager
  

• YARN-2070. Minor sub-task reported by Zhijie Shen and fixed by Robert Kanter
  DistributedShell publishes unfriendly user information to the timeline server
  

• YARN-2045. Major sub-task reported by Junping Du and fixed by Junping Du (nodemanager)
  Data persisted in NM should be versioned
  

• YARN-2035. Major sub-task reported by Jonathan Eagles and fixed by Jonathan Eagles
  FileSystemApplicationHistoryStore blocks RM and AHS while NN is in safemode
  

• YARN-2034. Minor bug reported by Jason Lowe and fixed by Chen He (nodemanager)
  Description for yarn.nodemanager.localizer.cache.target-size-mb is incorrect
  

• YARN-2033. Major sub-task reported by Vinod Kumar Vavilapalli and fixed by Zhijie Shen
  Merging generic-history into the Timeline Store
  

• YARN-2026. Major bug reported by Ashwin Shankar and fixed by Ashwin Shankar (scheduler)
  Fair scheduler: Consider only active queues for computing fairshare
  

• YARN-2013. Major sub-task reported by Zhijie Shen and fixed by Tsuyoshi OZAWA (nodemanager)
  The diagnostics is always the ExitCodeException stack when the container crashes
  

• YARN-2010. Blocker bug reported by bc Wong and fixed by Karthik Kambatla (resourcemanager)
  Handle app-recovery failures gracefully
  

• YARN-2008. Major sub-task reported by Chen He and fixed by Craig Welch
  CapacityScheduler may report incorrect queueMaxCap if there is hierarchy queue structure
  

• YARN-2001. Major sub-task reported by Jian He and fixed by Jian He (resourcemanager)
  Threshold for RM to accept requests from AM after failover
  

• YARN-1994. Major improvement reported by Arpit Agarwal and fixed by Craig Welch (nodemanager , resourcemanager , webapp)
  Expose YARN/MR endpoints on multiple interfaces
  

• YARN-1972. Major sub-task reported by Remus Rusanu and fixed by Remus Rusanu (nodemanager)
  Implement secure Windows Container Executor
  

• YARN-1964. Major new feature reported by Arun C Murthy and fixed by Abin Shahab
  Create Docker analog of the LinuxContainerExecutor in YARN
  

• YARN-1959. Major bug reported by Sandy Ryza and fixed by Anubhav Dhoot
  Fix headroom calculation in FairScheduler
  

• YARN-1954. Major improvement reported by Zhijie Shen and fixed by Tsuyoshi OZAWA (client)
  Add waitFor to AMRMClient(Async)
  

• YARN-1922. Major bug reported by Billie Rinaldi and fixed by Billie Rinaldi (nodemanager)
  Process group remains alive after container process is killed externally
  

• YARN-1919. Minor bug reported by Devaraj K and fixed by Tsuyoshi OZAWA (resourcemanager)
  Potential NPE in EmbeddedElectorService#stop
  

• YARN-1918. Trivial improvement reported by Devaraj K and fixed by Anandha L Ranganathan
  Typo in description and error message for 'yarn.resourcemanager.cluster-id'
  

• YARN-1915. Blocker sub-task reported by Hitesh Shah and fixed by Jason Lowe
  ClientToAMTokenMasterKey should be provided to AM at launch time
  

• YARN-1879. Critical sub-task reported by Jian He and fixed by Tsuyoshi OZAWA (resourcemanager)
  Mark Idempotent/AtMostOnce annotations to ApplicationMasterProtocol for RM fail over
  

• YARN-1857. Critical sub-task reported by Thomas Graves and fixed by Chen He (capacityscheduler)
  CapacityScheduler headroom doesn't account for other AM's running
  

• YARN-1796. Minor bug reported by Aaron T. Myers and fixed by Aaron T. Myers (nodemanager)
  container-executor shouldn't require o-r permissions
  

• YARN-1779. Blocker sub-task reported by Karthik Kambatla and fixed by Jian He (resourcemanager)
  Handle AMRMTokens across RM failover
  

• YARN-1769. Major improvement reported by Thomas Graves and fixed by Thomas Graves (capacityscheduler)
  CapacityScheduler: Improve reservations
  

• YARN-1712. Major sub-task reported by Carlo Curino and fixed by Carlo Curino (capacityscheduler , resourcemanager)
  Admission Control: plan follower
  

• YARN-1711. Major sub-task reported by Carlo Curino and fixed by Carlo Curino
  CapacityOverTimePolicy: a policy to enforce quotas over time for YARN-1709
  

• YARN-1710. Major sub-task reported by Carlo Curino and fixed by Carlo Curino (resourcemanager)
  Admission Control: agents to allocate reservation
  

• YARN-1709. Major sub-task reported by Carlo Curino and fixed by Subru Krishnan (resourcemanager)
  Admission Control: Reservation subsystem
  

• YARN-1708. Major sub-task reported by Carlo Curino and fixed by Subru Krishnan (resourcemanager)
  Add a public API to reserve resources (part of YARN-1051)
  

• YARN-1707. Major sub-task reported by Carlo Curino and fixed by Carlo Curino (capacityscheduler)
  Making the CapacityScheduler more dynamic
  

• YARN-1372. Major sub-task reported by Bikas Saha and fixed by Anubhav Dhoot (resourcemanager)
  Ensure all completed containers are reported to the AMs across RM restart
  

• YARN-1370. Major sub-task reported by Bikas Saha and fixed by Anubhav Dhoot (resourcemanager)
  Fair scheduler to re-populate container allocation state
  

• YARN-1367. Major sub-task reported by Bikas Saha and fixed by Anubhav Dhoot (resourcemanager)
  After restart NM should resync with the RM without killing containers
  

• YARN-1354. Major sub-task reported by Jason Lowe and fixed by Jason Lowe (nodemanager)
  Recover applications upon nodemanager restart
  

• YARN-1342. Major sub-task reported by Jason Lowe and fixed by Jason Lowe (nodemanager)
  Recover container tokens upon nodemanager restart
  

• YARN-1341. Major sub-task reported by Jason Lowe and fixed by Jason Lowe (nodemanager)
  Recover NMTokens upon nodemanager restart
  

• YARN-1337. Major sub-task reported by Jason Lowe and fixed by Jason Lowe (nodemanager)
  Recover containers upon nodemanager restart
  

• YARN-1326. Major sub-task reported by Tsuyoshi OZAWA and fixed by Tsuyoshi OZAWA
  RM should log using RMStore at startup time
  

• YARN-1250. Major sub-task reported by Vinod Kumar Vavilapalli and fixed by Zhijie Shen
  Generic history service should support application-acls
  

• YARN-1063. Major sub-task reported by Kyle Leckie and fixed by Remus Rusanu (nodemanager)
  Winutils needs ability to create task as domain user
  

• YARN-1051. Major improvement reported by Carlo Curino and fixed by Carlo Curino (capacityscheduler , resourcemanager , scheduler)
  YARN Admission Control/Planner: enhancing the resource allocation model with time.
  

• YARN-668. Blocker sub-task reported by Siddharth Seth and fixed by Junping Du
  TokenIdentifier serialization should consider Unknown fields
  

• YARN-611. Major sub-task reported by Chris Riccomini and fixed by Xuan Gong (resourcemanager)
  Add an AM retry count reset window to YARN RM
  

• YARN-415. Major new feature reported by Kendall Thrapp and fixed by Eric Payne (resourcemanager)
  Capture aggregate memory allocation at the app-level for chargeback
  

• YARN-90. Major sub-task reported by Ravi Gummadi and fixed by Varun Vasudev (nodemanager)
  NodeManager should identify failed disks becoming good again
  

• MAPREDUCE-6156. Blocker bug reported by Sidharta Seethana and fixed by Junping Du
  Fetcher - connect() doesn't handle connection refused correctly
  

• MAPREDUCE-6142. Critical sub-task reported by Zhijie Shen and fixed by Zhijie Shen
  Test failure in TestJobHistoryEventHandler and TestMRTimelineEventHandling
  

• MAPREDUCE-6126. Major bug reported by Junping Du and fixed by Junping Du
  (Rumen) Rumen tool returns error "ava.lang.IllegalArgumentException: JobBuilder.process(HistoryEvent): unknown event type"
  

• MAPREDUCE-6125. Major bug reported by Mit Desai and fixed by Mit Desai (test)
  TestContainerLauncherImpl sometimes fails
  

• MAPREDUCE-6123. Trivial bug reported by Chris Nauroth and fixed by Chris Nauroth (test)
  TestCombineFileInputFormat incorrectly starts 2 MiniDFSCluster instances.
  

• MAPREDUCE-6122. Trivial bug reported by Chris Nauroth and fixed by Chris Nauroth (test)
  TestLineRecordReader may fail due to test data files checked out of git with incorrect line endings.
  

• MAPREDUCE-6115. Minor test reported by Ted Yu and fixed by Binglin Chang
  TestPipeApplication#testSubmitter fails in trunk
  

• MAPREDUCE-6109. Trivial bug reported by Charles Lamb and fixed by Charles Lamb (distcp)
  Fix minor typo in distcp -p usage text
  

• MAPREDUCE-6104. Major bug reported by Mit Desai and fixed by Mit Desai
  TestJobHistoryParsing.testPartialJob fails in branch-2
  

• MAPREDUCE-6095. Major bug reported by Gera Shegalov and fixed by Gera Shegalov (applicationmaster , distributed-cache)
  Enable DistributedCache for uber-mode Jobs
  

• MAPREDUCE-6094. Minor bug reported by Sangjin Lee and fixed by Akira AJISAKA (test)
  TestMRCJCFileInputFormat.testAddInputPath() fails on trunk
  

• MAPREDUCE-6093. Trivial bug reported by Charles Lamb and fixed by Charles Lamb (distcp , documentation)
  minor distcp doc edits
  

• MAPREDUCE-6091. Major bug reported by Sangjin Lee and fixed by Sangjin Lee (client)
  YARNRunner.getJobStatus() fails with ApplicationNotFoundException if the job rolled off the RM view
  

• MAPREDUCE-6090. Major bug reported by Robert Kanter and fixed by Robert Kanter (client)
  mapred hsadmin getGroups fails to connect in some cases
  

• MAPREDUCE-6086. Major improvement reported by zhihai xu and fixed by zhihai xu (security)
  mapreduce.job.credentials.binary should allow all URIs
  

• MAPREDUCE-6075. Major bug reported by Jason Lowe and fixed by Jason Lowe (jobhistoryserver)
  HistoryServerFileSystemStateStore can create zero-length files
  

• MAPREDUCE-6073. Trivial bug reported by Tsuyoshi OZAWA and fixed by Tsuyoshi OZAWA (documentation)
  Description of mapreduce.job.speculative.slowtaskthreshold in mapred-default should be moved into description tags
  

• MAPREDUCE-6072. Minor improvement reported by Akira AJISAKA and fixed by Akira AJISAKA (documentation)
  Remove INSTALL document
  

• MAPREDUCE-6071. Trivial improvement reported by Tsuyoshi OZAWA and fixed by Tsuyoshi OZAWA (client)
  JobImpl#makeUberDecision doesn't log that Uber mode is disabled because of too much CPUs
  

• MAPREDUCE-6070. Trivial improvement reported by Tsuyoshi OZAWA and fixed by Tsuyoshi OZAWA (documentation)
  yarn.app.am.resource.mb/cpu-vcores affects uber mode but is not documented
  

• MAPREDUCE-6063. Major bug reported by zhihai xu and fixed by zhihai xu (mrv1 , mrv2)
  In sortAndSpill of MapTask.java, size is calculated wrongly when bufend < bufstart.
  

• MAPREDUCE-6052. Major bug reported by Junping Du and fixed by Junping Du
  Support overriding log4j.properties per job
  

• MAPREDUCE-6051. Trivial bug reported by Ray Chiang and fixed by Ray Chiang
  Fix typos in log messages
  

• MAPREDUCE-6048. Minor test reported by Ted Yu and fixed by Varun Vasudev
  TestJavaSerialization fails in trunk build
  

• MAPREDUCE-6044. Major bug reported by Zhijie Shen and fixed by Zhijie Shen (jobhistoryserver)
  Fully qualified intermediate done directory will break per-user dir creation on Windows
  

• MAPREDUCE-6036. Major bug reported by Mit Desai and fixed by chang li
  TestJobEndNotifier fails intermittently in branch-2
  

• MAPREDUCE-6032. Major bug reported by Benjamin Zhitomirsky and fixed by Benjamin Zhitomirsky (jobhistoryserver)
  Unable to check mapreduce job status if submitted using a non-default namenode
  

• MAPREDUCE-6029. Major bug reported by Ted Yu and fixed by Mit Desai
  TestCommitterEventHandler fails in trunk
  

• MAPREDUCE-6024. Critical improvement reported by zhaoyunjiong and fixed by zhaoyunjiong (mr-am , task)
  java.net.SocketTimeoutException in Fetcher caused jobs stuck for more than 1 hour
  

• MAPREDUCE-6022. Major bug reported by Jason Lowe and fixed by Jason Lowe
  map_input_file is missing from streaming job environment
  

• MAPREDUCE-6021. Major bug reported by Jason Lowe and fixed by Jason Lowe (mr-am)
  MR AM should have working directory in LD_LIBRARY_PATH
  

• MAPREDUCE-6019. Major bug reported by Xuan Gong and fixed by Craig Welch
  MapReduce changes for exposing YARN/MR endpoints on multiple interfaces.
  

• MAPREDUCE-6018. Major sub-task reported by Jonathan Eagles and fixed by Robert Kanter
  Create a framework specific config to enable timeline server
  

• MAPREDUCE-6014. Major bug reported by Mit Desai and fixed by Mit Desai
  New task status field in task attempts table can lead to an empty web page
  

• MAPREDUCE-6012. Major bug reported by Julien Serdaru and fixed by Wei Yan
  DBInputSplit creates invalid ranges on Oracle
  

• MAPREDUCE-6010. Major bug reported by Jason Lowe and fixed by Jason Lowe (jobhistoryserver)
  HistoryServerFileSystemStateStore fails to update tokens
  

• MAPREDUCE-5999. Minor bug reported by Akira AJISAKA and fixed by Akira AJISAKA (documentation)
  Fix dead link in InputFormat javadoc
  

• MAPREDUCE-5998. Minor bug reported by Akira AJISAKA and fixed by Akira AJISAKA (documentation)
  CompositeInputFormat javadoc is broken
  

• MAPREDUCE-5974. Major sub-task reported by Todd Lipcon and fixed by Todd Lipcon (task)
  Allow specifying multiple MapOutputCollectors with fallback
  

• MAPREDUCE-5971. Trivial improvement reported by Charles Lamb and fixed by Charles Lamb (distcp)
  Move the default options for distcp -p to DistCpOptionSwitch
  

• MAPREDUCE-5970. Minor improvement reported by Gera Shegalov and fixed by Gera Shegalov (applicationmaster , client)
  Provide a boolean switch to enable MR-AM profiling
  

• MAPREDUCE-5963. Major sub-task reported by Junping Du and fixed by Junping Du
  ShuffleHandler DB schema should be versioned with compatible/incompatible changes
  

• MAPREDUCE-5960. Major bug reported by Gera Shegalov and fixed by Gera Shegalov (client)
  JobSubmitter's check whether job.jar is local is incorrect with no authority in job jar path.
  

• MAPREDUCE-5958. Minor bug reported by Emilio Coppa and fixed by Emilio Coppa
  Wrong reduce task progress if map output is compressed
  

• MAPREDUCE-5957. Major bug reported by Sangjin Lee and fixed by Sangjin Lee
  AM throws ClassNotFoundException with job classloader enabled if custom output format/committer is used
  

• MAPREDUCE-5956. Blocker sub-task reported by Vinod Kumar Vavilapalli and fixed by Wangda Tan (applicationmaster , mrv2)
  MapReduce AM should not use maxAttempts to determine if this is the last retry
  

• MAPREDUCE-5950. Major bug reported by Yongjun Zhang and fixed by Akira AJISAKA (documentation)
  incorrect description in distcp2 document
  

• MAPREDUCE-5945. Minor sub-task reported by Akira AJISAKA and fixed by Akira AJISAKA (documentation)
  Update the description of GenericOptionsParser -jt option
  

• MAPREDUCE-5943. Minor improvement reported by Akira AJISAKA and fixed by Akira AJISAKA (documentation)
  Separate mapred commands from CommandsManual.apt.vm
  

• MAPREDUCE-5942. Minor sub-task reported by Akira AJISAKA and fixed by Akira AJISAKA (documentation)
  Remove MRv1 commands from CommandsManual.apt.vm
  

• MAPREDUCE-5933. Major sub-task reported by Zhijie Shen and fixed by Robert Kanter (mr-am)
  Enable MR AM to post history events to the timeline server
  

• MAPREDUCE-5931. Minor bug reported by Gera Shegalov and fixed by Gera Shegalov (test)
  Validate SleepJob command line parameters
  

• MAPREDUCE-5910. Major task reported by Rohith and fixed by Rohith (applicationmaster)
  MRAppMaster should handle Resync from RM instead of shutting down.
  

• MAPREDUCE-5906. Minor improvement reported by Akira AJISAKA and fixed by Akira AJISAKA
  Inconsistent configuration in property "mapreduce.reduce.shuffle.input.buffer.percent"
  

• MAPREDUCE-5891. Major sub-task reported by Jason Lowe and fixed by Junping Du
  Improved shuffle error handling across NM restarts
  

• MAPREDUCE-5885. Major bug reported by Jason Lowe and fixed by Chen He (test)
  build/test/test.mapred.spill causes release audit warnings
  

• MAPREDUCE-5878. Major bug reported by Sangjin Lee and fixed by Sangjin Lee (mrv2)
  some standard JDK APIs are not part of system classes defaults
  

• MAPREDUCE-5873. Major bug reported by Siqi Li and fixed by Siqi Li
  Shuffle bandwidth computation includes time spent waiting for maps
  

• MAPREDUCE-5866. Major test reported by Varun Vasudev and fixed by Varun Vasudev (client , test)
  TestFixedLengthInputFormat fails in windows
  

• MAPREDUCE-5831. Blocker bug reported by Zhijie Shen and fixed by Junping Du (client , mr-am)
  Old MR client is not compatible with new MR application
  

• MAPREDUCE-5796. Minor bug reported by Akira AJISAKA and fixed by Akira AJISAKA (documentation)
  Use current version of the archive name in DistributedCacheDeploy document
  

• MAPREDUCE-5756. Major bug reported by Jason Dere and fixed by Jason Dere
  CombineFileInputFormat.getSplits() including directories in its results
  

• MAPREDUCE-5597. Minor bug reported by Christopher Tubbs and fixed by Akira AJISAKA (client , documentation , job submission)
  Missing alternatives in javadocs for deprecated constructors in mapreduce.Job
  

• MAPREDUCE-5595. Trivial bug reported by Efe Gencer and fixed by Akira AJISAKA
  Typo in MergeManagerImpl.java
  

• MAPREDUCE-5542. Major bug reported by Jason Lowe and fixed by Rohith (client , mrv2)
  Killing a job just as it finishes can generate an NPE in client
  

• MAPREDUCE-5363. Minor bug reported by Sandy Ryza and fixed by Akira AJISAKA (mrv1 , mrv2)
  Fix doc and spelling for TaskCompletionEvent#getTaskStatus and getStatus
  

• MAPREDUCE-5279. Critical bug reported by Peng Zhang and fixed by Peng Zhang (mrv2 , scheduler)
  Jobs can deadlock if headroom is limited by cpu instead of memory
  

• MAPREDUCE-5130. Major improvement reported by Sandy Ryza and fixed by Ray Chiang (documentation)
  Add missing job config options to mapred-default.xml
  

• MAPREDUCE-4791. Minor improvement reported by Matt Lavin and fixed by Akira AJISAKA (documentation)
  Javadoc for KeyValueTextInputFormat should include default separator and how to change it
  

• MAPREDUCE-883. Minor improvement reported by Koji Noguchi and fixed by Akira AJISAKA (documentation , harchive)
  harchive: Document how to unarchive
  

• HDFS-7391. Blocker bug reported by Robert Kanter and fixed by Robert Kanter (webhdfs)
  Renable SSLv2Hello in HttpFS
  

• HDFS-7387. Critical bug reported by Brandon Li and fixed by Brandon Li (nfs)
  NFS may only do partial commit due to a race between COMMIT and write
  

• HDFS-7385. Blocker bug reported by jiangyu and fixed by jiangyu (namenode)
  ThreadLocal used in FSEditLog class causes FSImage permission mess up
  

• HDFS-7383. Major bug reported by Tsz Wo Nicholas Sze and fixed by Tsz Wo Nicholas Sze (datanode)
  DataNode.requestShortCircuitFdsForRead may throw NullPointerException
  

• HDFS-7382. Minor bug reported by Chris Nauroth and fixed by Chris Nauroth (datanode , security)
  DataNode in secure mode may throw NullPointerException if client connects before DataNode registers itself with NameNode.
  

• HDFS-7379. Minor bug reported by Xiaoyu Yao and fixed by Xiaoyu Yao (test)
  TestBalancer#testBalancerWithRamDisk creates test files incorrectly
  

• HDFS-7367. Major bug reported by Chris Nauroth and fixed by Chris Nauroth (hdfs-client)
  HDFS short-circuit read cannot negotiate shared memory slot and file descriptors when SASL is enabled on DataTransferProtocol.
  

• HDFS-7364. Minor bug reported by Tsz Wo Nicholas Sze and fixed by Tsz Wo Nicholas Sze (balancer & mover)
  Balancer always shows zero Bytes Already Moved
  

• HDFS-7359. Major bug reported by Chris Nauroth and fixed by Chris Nauroth (journal-node , namenode)
  NameNode in secured HA cluster fails to start if dfs.namenode.secondary.http-address cannot be interpreted as a network address.
  

• HDFS-7355. Trivial test reported by Chris Nauroth and fixed by Chris Nauroth (test)
  TestDataNodeVolumeFailure#testUnderReplicationAfterVolFailure fails on Windows, because we cannot deny access to the file owner.
  

• HDFS-7340. Major bug reported by Arpit Gupta and fixed by Jing Zhao (ha)
  make rollingUpgrade start/finalize idempotent
  

• HDFS-7334. Minor bug reported by Charles Lamb and fixed by Charles Lamb (test)
  Fix periodic failures of TestCheckpoint#testTooManyEditReplayFailures
  

• HDFS-7328. Trivial bug reported by Chris Nauroth and fixed by Chris Nauroth (test)
  TestTraceAdmin assumes Unix line endings.
  

• HDFS-7313. Major improvement reported by Chris Nauroth and fixed by Chris Nauroth (datanode , hdfs-client , security)
  Support optional configuration of AES cipher suite on DataTransferProtocol.
  

• HDFS-7309. Minor bug reported by Ravi Prakash and fixed by Colin Patrick McCabe
  XMLUtils.mangleXmlString doesn't seem to handle less than sign
  

• HDFS-7305. Minor bug reported by Arpit Gupta and fixed by Jing Zhao (webhdfs)
  NPE seen in wbhdfs FS while running SLive
  

• HDFS-7300. Critical bug reported by Kihwal Lee and fixed by Kihwal Lee
  The getMaxNodesPerRack() method in BlockPlacementPolicyDefault is flawed
  

• HDFS-7296. Minor bug reported by Jing Zhao and fixed by Jing Zhao
  HdfsConstants#MEMORY_STORAGE_POLICY_ID and HdfsConstants#MEMORY_STORAGE_POLICY_ID are missing in branch-2
  

• HDFS-7291. Major sub-task reported by Xiaoyu Yao and fixed by Xiaoyu Yao (datanode)
  Persist in-memory replicas with appropriate unbuffered copy API on POSIX and Windows
  

• HDFS-7287. Major bug reported by Ravi Prakash and fixed by Ravi Prakash
  The OfflineImageViewer (OIV) can output invalid XML depending on the filename
  

• HDFS-7276. Major improvement reported by Tsz Wo Nicholas Sze and fixed by Tsz Wo Nicholas Sze (hdfs-client)
  Limit the number of byte arrays used by DFSOutputStream
  

• HDFS-7274. Blocker bug reported by Robert Kanter and fixed by Robert Kanter (webhdfs)
  Disable SSLv3 in HttpFS
  

• HDFS-7260. Minor improvement reported by Tsz Wo Nicholas Sze and fixed by Tsz Wo Nicholas Sze (hdfs-client)
  Make DFSOutputStream.MAX_PACKETS configurable
  

• HDFS-7259. Major bug reported by Brandon Li and fixed by Brandon Li (nfs)
  Unresponseive NFS mount point due to deferred COMMIT response
  

• HDFS-7243. Major bug reported by Yi Liu and fixed by Charles Lamb (encryption , namenode)
  HDFS concat operation should not be allowed in Encryption Zone
  

• HDFS-7237. Minor bug reported by Tsz Wo Nicholas Sze and fixed by Tsz Wo Nicholas Sze (namenode)
  namenode -rollingUpgrade throws ArrayIndexOutOfBoundsException
  

• HDFS-7236. Major bug reported by Yongjun Zhang and fixed by Yongjun Zhang
  Fix TestOpenFilesWithSnapshot#testOpenFilesWithMultipleSnapshots
  

• HDFS-7233. Major improvement reported by Rushabh S Shah and fixed by Rushabh S Shah (namenode)
  NN logs unnecessary org.apache.hadoop.hdfs.protocol.UnresolvedPathException
  

• HDFS-7230. Major improvement reported by Tsz Wo Nicholas Sze and fixed by Tsz Wo Nicholas Sze (documentation)
  Add rolling downgrade documentation
  

• HDFS-7228. Major improvement reported by Jing Zhao and fixed by Jing Zhao
  Add an SSD policy into the default BlockStoragePolicySuite
  

• HDFS-7226. Major bug reported by Yongjun Zhang and fixed by Yongjun Zhang (ha)
  TestDNFencing.testQueueingWithAppend failed often in latest test
  

• HDFS-7221. Minor bug reported by Charles Lamb and fixed by Charles Lamb (test)
  TestDNFencingWithReplication fails consistently
  

• HDFS-7218. Minor bug reported by Charles Lamb and fixed by Charles Lamb (namenode)
  FSNamesystem ACL operations should write to audit log on failure
  

• HDFS-7217. Major improvement reported by Kihwal Lee and fixed by Kihwal Lee
  Better batching of IBRs
  

• HDFS-7215. Minor improvement reported by Brandon Li and fixed by Brandon Li (nfs)
  Add JvmPauseMonitor to NFS gateway
  

• HDFS-7208. Major bug reported by Ming Ma and fixed by Ming Ma (namenode)
  NN doesn't schedule replication when a DN storage fails
  

• HDFS-7199. Critical bug reported by Jason Lowe and fixed by Rushabh S Shah (hdfs-client)
  DFSOutputStream should not silently drop data if DataStreamer crashes with an unchecked exception
  

• HDFS-7195. Major improvement reported by Yi Liu and fixed by Chris Nauroth (documentation , security)
  Update user doc of secure mode about Datanodes don't require root or jsvc
  

• HDFS-7185. Major bug reported by Colin Patrick McCabe and fixed by Jing Zhao (namenode)
  The active NameNode will not accept an fsimage sent from the standby during rolling upgrade
  

• HDFS-7181. Critical sub-task reported by Andrew Wang and fixed by Andrew Wang (encryption)
  Remove incorrect precondition check on key length in FileEncryptionInfo
  

• HDFS-7180. Critical bug reported by Eric Zhiqiang Ma and fixed by Brandon Li (nfs)
  NFSv3 gateway frequently gets stuck due to GC
  

• HDFS-7179. Critical sub-task reported by Andrew Wang and fixed by Andrew Wang (encryption)
  DFSClient should instantiate a KeyProvider, not a KeyProviderCryptoExtension
  

• HDFS-7178. Major bug reported by Arpit Agarwal and fixed by Arpit Agarwal (test)
  Additional unit test for replica write with full disk
  

• HDFS-7176. Minor bug reported by Colin Patrick McCabe and fixed by Colin Patrick McCabe (namenode)
  The namenode usage message doesn't include "-rollingupgrade started"
  

• HDFS-7172. Trivial bug reported by Chris Nauroth and fixed by Chris Nauroth (test)
  Test data files may be checked out of git with incorrect line endings, causing test failures in TestHDFSCLI.
  

• HDFS-7171. Major sub-task reported by Arpit Agarwal and fixed by Arpit Agarwal (datanode)
  Fix Jenkins failures in HDFS-6581 branch
  

• HDFS-7169. Minor bug reported by Tsz Wo Nicholas Sze and fixed by Tsz Wo Nicholas Sze (build)
  Fix a findbugs warning in ReplaceDatanodeOnFailure
  

• HDFS-7167. Major bug reported by Prabushankar Chinnasamy and fixed by Jing Zhao (balancer & mover)
  NPE while running Mover if the given path is for a file
  

• HDFS-7162. Major bug reported by Chengbing Liu and fixed by Chengbing Liu (fuse-dfs)
  Wrong path when deleting through fuse-dfs a file which already exists in trash
  

• HDFS-7159. Major sub-task reported by Arpit Agarwal and fixed by Arpit Agarwal (namenode)
  Use block storage policy to set lazy persist preference
  

• HDFS-7158. Major improvement reported by Haohui Mai and fixed by Haohui Mai
  Reduce the memory usage of WebImageViewer
  

• HDFS-7157. Major sub-task reported by Lei (Eddy) Xu and fixed by Lei (Eddy) Xu (datanode)
  Using Time.now() for recording start/end time of reconfiguration tasks
  

• HDFS-7156. Major bug reported by Konstantin Shvachko and fixed by Masahiro Yamaguchi (documentation)
  Fsck documentation is outdated.
  

• HDFS-7155. Major sub-task reported by Arpit Agarwal and fixed by Arpit Agarwal (datanode)
  Bugfix in createLocatedFileStatus caused by bad merge
  

• HDFS-7154. Major sub-task reported by Lei (Eddy) Xu and fixed by Lei (Eddy) Xu (datanode)
  Fix returning value of starting reconfiguration task
  

• HDFS-7153. Major improvement reported by Arpit Agarwal and fixed by Arpit Agarwal (namenode)
  Add storagePolicy to NN edit log during file creation
  

• HDFS-7148. Major bug reported by Andrew Wang and fixed by Andrew Wang (encryption)
  TestEncryptionZones#testIsEncryptedMethod fails on branch-2 after archival storage merge
  

• HDFS-7147. Blocker bug reported by Tsz Wo Nicholas Sze and fixed by Tsz Wo Nicholas Sze (documentation)
  Update archival storage user documentation
  

• HDFS-7144. Minor sub-task reported by Tsz Wo Nicholas Sze and fixed by Tsz Wo Nicholas Sze (datanode)
  Fix findbugs warnings in RamDiskReplicaTracker
  

• HDFS-7143. Major sub-task reported by Tsz Wo Nicholas Sze and fixed by Tsz Wo Nicholas Sze (datanode)
  Fix findbugs warnings in HDFS-6581 branch
  

• HDFS-7140. Minor sub-task reported by Jing Zhao and fixed by Jing Zhao (hdfs-client , namenode)
  Add a tool to list all the existing block storage policies
  

• HDFS-7139. Minor sub-task reported by Zhe Zhang and fixed by Zhe Zhang
  Unit test for creating encryption zone on root path
  

• HDFS-7138. Major sub-task reported by Charles Lamb and fixed by Charles Lamb (namenode)
  Fix hftp to work with encryption
  

• HDFS-7132. Minor bug reported by Charles Lamb and fixed by Charles Lamb (namenode)
  hdfs namenode -metadataVersion command does not honor configured name dirs
  

• HDFS-7131. Major bug reported by Jing Zhao and fixed by Jing Zhao (qjm)
  During HA upgrade, JournalNode should create a new committedTxnId file in the current directory
  

• HDFS-7130. Major bug reported by Chris Nauroth and fixed by Chris Nauroth (test)
  TestDataTransferKeepalive fails intermittently on Windows.
  

• HDFS-7129. Major sub-task reported by Arpit Agarwal and fixed by Xiaoyu Yao (datanode)
  Metrics to track usage of memory for writes
  

• HDFS-7128. Major improvement reported by Ming Ma and fixed by Ming Ma (namenode)
  Decommission slows way down when it gets towards the end
  

• HDFS-7127. Major bug reported by Chris Nauroth and fixed by Chris Nauroth (test)
  TestLeaseRecovery leaks MiniDFSCluster instances.
  

• HDFS-7126. Minor test reported by Xiaoyu Yao and fixed by Xiaoyu Yao (security , test)
  TestEncryptionZonesWithHA assumes Unix path separator for KMS key store path
  

• HDFS-7124. Minor sub-task reported by Charles Lamb and fixed by Charles Lamb (namenode)
  Remove EncryptionZoneManager.NULL_EZ
  

• HDFS-7122. Blocker bug reported by Jeff Buell and fixed by Andrew Wang (namenode)
  Use of ThreadLocal<Random> results in poor block placement
  

• HDFS-7119. Minor sub-task reported by Chris Nauroth and fixed by Chris Nauroth (journal-node)
  Split error checks in AtomicFileOutputStream#close into separate conditions to improve diagnostics.
  

• HDFS-7118. Major sub-task reported by Chris Nauroth and fixed by Chris Nauroth (journal-node , namenode)
  Improve diagnostics on storage directory rename operations by using NativeIO#renameTo in Storage#rename.
  

• HDFS-7115. Major test reported by Xiaoyu Yao and fixed by Xiaoyu Yao (encryption)
  TestEncryptionZones assumes Unix path separator for KMS key store path
  

• HDFS-7112. Major sub-task reported by Arpit Agarwal and fixed by Xiaoyu Yao (datanode)
  LazyWriter should use either async IO or one thread per physical disk
  

• HDFS-7111. Trivial bug reported by Chris Nauroth and fixed by Chris Nauroth (test)
  TestSafeMode assumes Unix line endings in safe mode tip.
  

• HDFS-7110. Minor bug reported by Chris Nauroth and fixed by Chris Nauroth (test)
  Skip tests related to short-circuit read on platforms that do not currently implement short-circuit read.
  

• HDFS-7109. Minor bug reported by Chris Nauroth and fixed by Chris Nauroth (test)
  TestDataStorage does not release file locks between tests.
  

• HDFS-7108. Major sub-task reported by Arpit Agarwal and fixed by Arpit Agarwal (test)
  Fix unit test failures in SimulatedFsDataset
  

• HDFS-7107. Trivial bug reported by Chris Nauroth and fixed by Chris Nauroth (ha)
  Avoid Findbugs warning for synchronization on AbstractNNFailoverProxyProvider#fallbackToSimpleAuth.
  

• HDFS-7106. Major bug reported by Chris Nauroth and fixed by Chris Nauroth (datanode)
  Reconfiguring DataNode volumes does not release the lock files in removed volumes.
  

• HDFS-7105. Minor bug reported by Ray Chiang and fixed by Ray Chiang (test)
  Fix TestJournalNode#testFailToStartWithBadConfig to match log output change
  

• HDFS-7104. Minor bug reported by Zhe Zhang and fixed by Zhe Zhang
  Fix and clarify INodeInPath getter functions
  

• HDFS-7100. Major sub-task reported by Arpit Agarwal and fixed by Arpit Agarwal (datanode)
  Make eviction scheme pluggable
  

• HDFS-7096. Minor bug reported by Charles Lamb and fixed by Charles Lamb (test)
  Fix TestRpcProgramNfs3 to use DFS_ENCRYPTION_KEY_PROVIDER_URI
  

• HDFS-7095. Minor sub-task reported by Tsz Wo Nicholas Sze and fixed by Jing Zhao (test)
  TestStorageMover often fails in Jenkins
  

• HDFS-7093. Major improvement reported by Arpit Agarwal and fixed by Arpit Agarwal (namenode)
  Add config key to restrict setStoragePolicy
  

• HDFS-7091. Minor sub-task reported by Arpit Agarwal and fixed by Arpit Agarwal (namenode , test)
  Add forwarding constructor for INodeFile for existing callers
  

  Thanks Nicholas! Revised title and committed to the feature branch.

• HDFS-7090. Major sub-task reported by Arpit Agarwal and fixed by Xiaoyu Yao (datanode)
  Use unbuffered writes when persisting in-memory replicas
  

• HDFS-7084. Minor sub-task reported by Xiaoyu Yao and fixed by Xiaoyu Yao (datanode)
  FsDatasetImpl#copyBlockFiles debug log can be improved
  

• HDFS-7081. Major sub-task reported by Jing Zhao and fixed by Jing Zhao (balancer & mover , namenode)
  Add new DistributedFileSystem API for getting all the existing storage policies
  

• HDFS-7080. Major sub-task reported by Arpit Agarwal and fixed by Arpit Agarwal (test)
  Fix finalize and upgrade unit test failures
  

• HDFS-7079. Major sub-task reported by Arpit Agarwal and fixed by Arpit Agarwal (test)
  Few more unit test fixes for HDFS-6581
  

• HDFS-7078. Major sub-task reported by Andrew Wang and fixed by Andrew Wang (encryption)
  Fix listEZs to work correctly with snapshots
  

• HDFS-7077. Major sub-task reported by Andrew Wang and fixed by Andrew Wang (encryption)
  Separate CipherSuite from crypto protocol version
  

• HDFS-7075. Major bug reported by Colin Patrick McCabe and fixed by Colin Patrick McCabe
  hadoop-fuse-dfs fails because it cannot find JavaKeyStoreProvider$Factory
  

• HDFS-7073. Major bug reported by Chris Nauroth and fixed by Chris Nauroth (datanode , hdfs-client , security)
  Allow falling back to a non-SASL connection on DataTransferProtocol in several edge cases.
  

• HDFS-7066. Minor sub-task reported by Xiaoyu Yao and fixed by Xiaoyu Yao (datanode)
  LazyWriter#evictBlocks misses a null check for replicaState
  

• HDFS-7065. Critical bug reported by Kihwal Lee and fixed by Kihwal Lee (datanode)
  Pipeline close recovery race can cause block corruption
  

• HDFS-7064. Major sub-task reported by Arpit Agarwal and fixed by Xiaoyu Yao (test)
  Fix unit test failures in HDFS-6581 branch
  

• HDFS-7061. Minor sub-task reported by Stephen Chu and fixed by Stephen Chu (encryption , test)
  Add test to verify encryption zone creation after NameNode restart without saving namespace
  

• HDFS-7059. Minor improvement reported by Rushabh S Shah and fixed by Rushabh S Shah
  HAadmin transtionToActive with forceActive option can show confusing message.
  

• HDFS-7051. Minor test reported by Chris Nauroth and fixed by Chris Nauroth (datanode , test)
  TestDataNodeRollingUpgrade#isBlockFileInPrevious assumes Unix file path separator.
  

• HDFS-7049. Minor bug reported by Juan Yu and fixed by Juan Yu (test)
  TestByteRangeInputStream.testPropagatedClose fails and throw NPE on branch-2
  

• HDFS-7047. Major sub-task reported by Andrew Wang and fixed by Colin Patrick McCabe (encryption)
  Expose FileStatus#isEncrypted in libhdfs
  

• HDFS-7046. Critical bug reported by Daryn Sharp and fixed by Kihwal Lee (namenode)
  HA NN can NPE upon transition to active
  

  Thanks for the reviews, gentlemen. It's been committed to trunk and branch-2.

• HDFS-7045. Critical bug reported by Yi Liu and fixed by Yi Liu (namenode)
  Fix NameNode deadlock when opening file under /.reserved path
  

• HDFS-7042. Blocker bug reported by Chris Nauroth and fixed by Chris Nauroth (journal-node)
  Upgrade fails for Windows HA cluster due to file locks held during rename in JournalNode.
  

• HDFS-7032. Major sub-task reported by Stephen Chu and fixed by Charles Lamb (encryption , webhdfs)
  Add WebHDFS support for reading and writing to encryption zones
  

• HDFS-7025. Major test reported by Xiaoyu Yao and fixed by Xiaoyu Yao (encryption)
  HDFS Credential Provider related Unit Test Failure
  

• HDFS-7006. Major test reported by Alejandro Abdelnur and fixed by Alejandro Abdelnur (security , test)
  Test encryption zones with KMS
  

• HDFS-7005. Critical bug reported by Daryn Sharp and fixed by Daryn Sharp (hdfs-client)
  DFS input streams do not timeout
  

• HDFS-7004. Major sub-task reported by Andrew Wang and fixed by Andrew Wang (encryption)
  Update KeyProvider instantiation to create by URI
  

• HDFS-7003. Major sub-task reported by Stephen Chu and fixed by Charles Lamb (encryption , nfs)
  Add NFS Gateway support for reading and writing to encryption zones
  

• HDFS-7001. Minor bug reported by Masatake Iwasaki and fixed by Masatake Iwasaki
  Tests in TestTracing should not depend on the order of execution
  

• HDFS-6996. Major bug reported by Jing Zhao and fixed by Jing Zhao (snapshots)
  SnapshotDiff report can hit IndexOutOfBoundsException when there are nested renamed directory/file
  

• HDFS-6995. Major bug reported by Vinayakumar B and fixed by Vinayakumar B (namenode)
  Block should be placed in the client's 'rack-local' node if 'client-local' node is not available
  

• HDFS-6991. Major sub-task reported by Arpit Agarwal and fixed by Arpit Agarwal (datanode , namenode)
  Notify NN of evicted block before deleting it from RAM disk
  

• HDFS-6990. Major sub-task reported by Xiaoyu Yao and fixed by Xiaoyu Yao (datanode)
  Add unit test for evict/delete RAM_DISK block with open handle
  

• HDFS-6988. Major sub-task reported by Arpit Agarwal and fixed by Xiaoyu Yao (datanode)
  Improve HDFS-6581 eviction configuration
  

• HDFS-6987. Major sub-task reported by Andrew Wang and fixed by Zhe Zhang (encryption)
  Move CipherSuite xattr information up to the encryption zone root
  

• HDFS-6986. Major sub-task reported by Alejandro Abdelnur and fixed by Zhe Zhang (security)
  DistributedFileSystem must get delegation tokens from configured KeyProvider
  

• HDFS-6981. Major bug reported by James Thomas and fixed by Arpit Agarwal (datanode)
  Fix DN upgrade with layout version change
  

• HDFS-6979. Minor bug reported by Remus Rusanu and fixed by Chris Nauroth (hdfs-client)
  hdfs.dll does not produce .pdb files
  

• HDFS-6978. Major sub-task reported by Arpit Agarwal and fixed by Arpit Agarwal (datanode)
  Directory scanner should correctly reconcile blocks on RAM disk
  

• HDFS-6977. Major sub-task reported by Nathan Yao and fixed by Arpit Agarwal (datanode)
  Delete all copies when a block is deleted from the block space
  

• HDFS-6972. Major bug reported by Yongjun Zhang and fixed by Yongjun Zhang
  TestRefreshUserMappings.testRefreshSuperUserGroupsConfiguration doesn't decode url correctly
  

• HDFS-6970. Major sub-task reported by Andrew Wang and fixed by Andrew Wang (encryption)
  Move startFile EDEK retries to the DFSClient
  

• HDFS-6966. Major sub-task reported by Stephen Chu and fixed by Stephen Chu (encryption)
  Add additional unit tests for encryption zones
  

• HDFS-6965. Major bug reported by Daryn Sharp and fixed by Rushabh S Shah (namenode)
  NN continues to issue block locations for DNs with full disks
  

• HDFS-6960. Major sub-task reported by Arpit Agarwal and fixed by Arpit Agarwal (datanode , test)
  Bugfix in LazyWriter, fix test case and some refactoring
  

• HDFS-6959. Minor new feature reported by Kevin Odell and fixed by Yongjun Zhang
  Make the HDFS home directory location customizable.
  

• HDFS-6956. Major sub-task reported by Colin Patrick McCabe and fixed by Colin Patrick McCabe (datanode , namenode)
  Allow dynamically changing the tracing level in Hadoop servers
  

• HDFS-6954. Major bug reported by Allen Wittenauer and fixed by Charles Lamb (encryption)
  With crypto, no native lib systems are too verbose
  

• HDFS-6951. Major sub-task reported by Stephen Chu and fixed by Charles Lamb (encryption)
  Correctly persist raw namespace xattrs to edit log and fsimage
  

• HDFS-6950. Major sub-task reported by Xiaoyu Yao and fixed by Xiaoyu Yao
  Add Additional unit tests for HDFS-6581
  

• HDFS-6948. Major bug reported by Daryn Sharp and fixed by Eric Payne
  DN rejects blocks if it has older UC block
  

• HDFS-6943. Minor improvement reported by Ming Ma and fixed by Ming Ma (namenode)
  Improve NN allocateBlock log to include replicas' datanode IPs
  

• HDFS-6942. Trivial bug reported by Ray Chiang and fixed by Ray Chiang
  Fix typos in log messages
  

• HDFS-6934. Major sub-task reported by Arpit Agarwal and fixed by Chris Nauroth (datanode , hdfs-client)
  Move checksum computation off the hot path when writing to RAM disk
  

• HDFS-6932. Major sub-task reported by Arpit Agarwal and fixed by Xiaoyu Yao (datanode)
  Balancer and Mover tools should ignore replicas on RAM_DISK
  

• HDFS-6931. Major sub-task reported by Arpit Agarwal and fixed by Arpit Agarwal (datanode)
  Move lazily persisted replicas to finalized directory on DN startup
  

• HDFS-6930. Major sub-task reported by Arpit Agarwal and fixed by Arpit Agarwal (datanode)
  Improve replica eviction from RAM disk
  

• HDFS-6929. Major sub-task reported by Arpit Agarwal and fixed by Arpit Agarwal (datanode)
  NN periodically unlinks lazy persist files with missing replicas from namespace
  

• HDFS-6928. Major sub-task reported by Tassapol Athiapinya and fixed by Arpit Agarwal (datanode)
  'hdfs put' command should accept lazyPersist flag for testing
  

• HDFS-6927. Major sub-task reported by Arpit Agarwal and fixed by Arpit Agarwal (datanode)
  Add unit tests
  

• HDFS-6926. Major sub-task reported by Arpit Agarwal and fixed by Arpit Agarwal (datanode)
  DN support for saving replicas to persistent storage and evicting in-memory replicas
  

• HDFS-6925. Major sub-task reported by Arpit Agarwal and fixed by Arpit Agarwal (datanode)
  DataNode should attempt to place replicas on transient storage first if lazyPersist flag is received
  

• HDFS-6924. Major sub-task reported by Arpit Agarwal and fixed by Arpit Agarwal (datanode)
  Add new RAM_DISK storage type
  

• HDFS-6923. Major sub-task reported by Arpit Agarwal and fixed by Arpit Agarwal (datanode)
  Propagate LazyPersist flag to DNs via DataTransferProtocol
  

• HDFS-6921. Major sub-task reported by Arpit Agarwal and fixed by Arpit Agarwal
  Add LazyPersist flag to FileStatus
  

• HDFS-6912. Minor bug reported by Gopal V and fixed by Colin Patrick McCabe (caching)
  SharedFileDescriptorFactory should not allocate sparse files
  

• HDFS-6908. Critical bug reported by Juan Yu and fixed by Juan Yu (snapshots)
  incorrect snapshot directory diff generated by snapshot deletion
  

• HDFS-6905. Blocker bug reported by Allen Wittenauer and fixed by Charles Lamb
  fs-encryption merge triggered release audit failures
  

• HDFS-6904. Critical bug reported by Varun Vasudev and fixed by Jitendra Nath Pandey (webhdfs)
  YARN unable to renew delegation token fetched via webhdfs due to incorrect service port
  

• HDFS-6902. Minor bug reported by Ted Yu and fixed by Tsuyoshi OZAWA
  FileWriter should be closed in finally block in BlockReceiver#receiveBlock()
  

• HDFS-6899. Major improvement reported by Arpit Agarwal and fixed by Arpit Agarwal (datanode , test)
  Allow changing MiniDFSCluster volumes per DN and capacity per volume
  

• HDFS-6894. Major sub-task reported by Brandon Li and fixed by Brandon Li (nfs)
  Add XDR parser method for each NFS response
  

• HDFS-6892. Major sub-task reported by Brandon Li and fixed by Brandon Li (nfs)
  Add XDR packaging method for each NFS request
  

• HDFS-6890. Major bug reported by Brandon Li and fixed by Brandon Li (nfs)
  NFS readdirplus doesn't return dotdot attributes
  

• HDFS-6886. Critical improvement reported by Yi Liu and fixed by Yi Liu (namenode)
  Use single editlog record for creating file + overwrite.
  

• HDFS-6880. Major sub-task reported by Masatake Iwasaki and fixed by Masatake Iwasaki
  Adding tracing to DataNode data transfer protocol
  

• HDFS-6879. Major sub-task reported by Masatake Iwasaki and fixed by Masatake Iwasaki
  Adding tracing to Hadoop RPC
  

• HDFS-6878. Minor test reported by Tsz Wo Nicholas Sze and fixed by Arpit Agarwal (test)
  Change MiniDFSCluster to support StorageType configuration for individual directories
  

• HDFS-6870. Major bug reported by Yi Liu and fixed by Yi Liu (namenode)
  Blocks and INodes could leak for Rename with overwrite flag
  

• HDFS-6868. Major bug reported by Allen Wittenauer and fixed by Brandon Li (documentation , nfs)
  portmap and nfs3 are documented as hadoop commands instead of hdfs
  

• HDFS-6865. Major sub-task reported by James Thomas and fixed by James Thomas (hdfs-client , performance)
  Byte array native checksumming on client side (HDFS changes)
  

• HDFS-6862. Major bug reported by Arpit Agarwal and fixed by Xiaoyu Yao (test)
  Add missing timeout annotations to tests
  

• HDFS-6858. Minor improvement reported by Benoy Antony and fixed by Benoy Antony (security)
  Allow dfs.data.transfer.saslproperties.resolver.class default to hadoop.security.saslproperties.resolver.class
  

• HDFS-6851. Major sub-task reported by Charles Lamb and fixed by Charles Lamb (namenode , security)
  Refactor EncryptionZoneWithId and EncryptionZone
  

• HDFS-6850. Minor improvement reported by Zhe Zhang and fixed by Zhe Zhang (nfs)
  Move NFS out of order write unit tests into TestWrites class
  

• HDFS-6849. Major improvement reported by Alejandro Abdelnur and fixed by Alejandro Abdelnur (security)
  Replace HttpFS custom proxyuser handling with common implementation
  

• HDFS-6848. Minor bug reported by Ted Yu and fixed by Xiaoyu Yao
  Lack of synchronization on access to datanodeUuid in DataStorage#format()
  

• HDFS-6843. Major sub-task reported by Charles Lamb and fixed by Charles Lamb (namenode , security)
  Create FileStatus isEncrypted() method
  

• HDFS-6840. Critical bug reported by Jason Lowe and fixed by Andrew Wang
  Clients are always sent to the same datanode when read is off rack
  

• HDFS-6838. Minor bug reported by Jing Zhao and fixed by Jing Zhao (namenode)
  Code cleanup for unnecessary INode replacement
  

• HDFS-6837. Minor improvement reported by Tsz Wo Nicholas Sze and fixed by Tsz Wo Nicholas Sze (balancer & mover)
  Code cleanup for Balancer and Dispatcher
  

• HDFS-6836. Major improvement reported by Gopal V and fixed by Nathan Yao (datanode)
  HDFS INFO logging is verbose & uses file appenders
  

• HDFS-6831. Minor bug reported by Akira AJISAKA and fixed by Xiaoyu Yao
  Inconsistency between 'hdfs dfsadmin' and 'hdfs dfsadmin -help'
  

• HDFS-6830. Major bug reported by Arpit Agarwal and fixed by Arpit Agarwal (namenode)
  BlockInfo.addStorage fails when DN changes the storage for a block replica
  

• HDFS-6829. Minor bug reported by zhaoyunjiong and fixed by zhaoyunjiong (tools)
  DFSAdmin refreshSuperUserGroupsConfiguration failed in security cluster
  

• HDFS-6828. Major improvement reported by Tsz Wo Nicholas Sze and fixed by Tsz Wo Nicholas Sze (balancer & mover)
  Separate block replica dispatching from Balancer
  

• HDFS-6825. Major bug reported by Yongjun Zhang and fixed by Yongjun Zhang (namenode)
  Edit log corruption due to delayed block removal
  

• HDFS-6823. Minor bug reported by Allen Wittenauer and fixed by Allen Wittenauer (namenode)
  dfs.web.authentication.kerberos.principal shows up in logs for insecure HDFS
  

• HDFS-6812. Minor improvement reported by Tsz Wo Nicholas Sze and fixed by Tsz Wo Nicholas Sze (namenode)
  Remove addBlock and replaceBlock from DatanodeDescriptor
  

• HDFS-6810. Minor bug reported by Ted Yu and fixed by Tsz Wo Nicholas Sze (namenode)
  StorageReport array is initialized with wrong size in DatanodeDescriptor#getStorageReports
  

• HDFS-6809. Minor improvement reported by Tsz Wo Nicholas Sze and fixed by Tsz Wo Nicholas Sze (balancer & mover)
  Move some Balancer's inner classes to standalone classes
  

• HDFS-6808. Major sub-task reported by Lei (Eddy) Xu and fixed by Lei (Eddy) Xu (datanode)
  Add command line option to ask DataNode reload configuration.
  

• HDFS-6802. Major bug reported by Akira AJISAKA and fixed by Akira AJISAKA (test)
  Some tests in TestDFSClientFailover are missing @Test annotation
  

• HDFS-6800. Major improvement reported by Colin Patrick McCabe and fixed by James Thomas (datanode)
  Support Datanode layout changes with rolling upgrade
  

• HDFS-6799. Minor bug reported by Megasthenis Asteris and fixed by Megasthenis Asteris (datanode , test)
  The invalidate method in SimulatedFSDataset.java failed to remove (invalidate) blocks from the file system.
  

• HDFS-6798. Major improvement reported by Benoy Antony and fixed by Benoy Antony (balancer & mover)
  Add test case for incorrect data node condition during balancing
  

• HDFS-6797. Major bug reported by Benoy Antony and fixed by Benoy Antony (datanode)
  DataNode logs wrong layoutversion during upgrade
  

• HDFS-6796. Minor improvement reported by Benoy Antony and fixed by Benoy Antony (balancer & mover)
  Improving the argument check during balancer command line parsing
  

• HDFS-6794. Minor improvement reported by Arpit Agarwal and fixed by Arpit Agarwal (namenode)
  Update BlockManager methods to use DatanodeStorageInfo where possible
  

• HDFS-6791. Major bug reported by Ming Ma and fixed by Ming Ma
  A block could remain under replicated if all of its replicas are on decommissioned nodes
  

• HDFS-6790. Major bug reported by Larry McCay and fixed by Larry McCay
  DFSUtil Should Use configuration.getPassword for SSL passwords
  

• HDFS-6789. Major bug reported by Rushabh S Shah and fixed by Akira AJISAKA (test)
  TestDFSClientFailover.testFileContextDoesntDnsResolveLogicalURI and TestDFSClientFailover.testDoesntDnsResolveLogicalURI failing on jdk7
  

• HDFS-6788. Major bug reported by Yongjun Zhang and fixed by Yongjun Zhang (datanode)
  Improve synchronization in BPOfferService with read write lock
  

• HDFS-6787. Major bug reported by Yi Liu and fixed by Yi Liu (namenode)
  Remove duplicate code in FSDirectory#unprotectedConcat
  

• HDFS-6783. Major bug reported by Yi Liu and fixed by Yi Liu (caching)
  Fix HDFS CacheReplicationMonitor rescan logic
  

• HDFS-6781. Major improvement reported by Akira AJISAKA and fixed by Akira AJISAKA (documentation)
  Separate HDFS commands from CommandsManual.apt.vm
  

• HDFS-6779. Minor improvement reported by Allen Wittenauer and fixed by Sasaki Toru (scripts)
  Add missing version subcommand for hdfs
  

• HDFS-6778. Major bug reported by Charles Lamb and fixed by Charles Lamb
  The extended attributes javadoc should simply refer to the user docs
  

• HDFS-6777. Major sub-task reported by James Thomas and fixed by James Thomas (qjm)
  Supporting consistent edit log reads when in-progress edit log segments are included
  

• HDFS-6776. Major bug reported by Yongjun Zhang and fixed by Yongjun Zhang
  Using distcp to copy data between insecure and secure cluster via webdhfs doesn't work
  

• HDFS-6774. Major sub-task reported by Lei (Eddy) Xu and fixed by Lei (Eddy) Xu (datanode)
  Make FsDataset and DataStore support removing volumes.
  

• HDFS-6773. Major improvement reported by Daryn Sharp and fixed by Stephen Chu (namenode)
  MiniDFSCluster should skip edit log fsync by default
  

• HDFS-6772. Major improvement reported by Ming Ma and fixed by Ming Ma
  Get DN storages out of blockContentsStale state faster after NN restarts
  

• HDFS-6768. Major bug reported by Arpit Agarwal and fixed by Arpit Agarwal (test)
  Fix a few unit tests that use hard-coded port numbers
  

• HDFS-6758. Major improvement reported by Arpit Agarwal and fixed by Arpit Agarwal (datanode , hdfs-client)
  block writer should pass the expected block size to DataXceiverServer
  

• HDFS-6755. Major improvement reported by Mit Desai and fixed by Mit Desai
  There is an unnecessary sleep in the code path where DFSOutputStream#close gives up its attempt to contact the namenode
  

• HDFS-6754. Major bug reported by Mit Desai and fixed by Mit Desai
  TestNamenodeCapacityReport.testXceiverCount may sometimes fail due to lack of retry
  

• HDFS-6750. Major sub-task reported by Colin Patrick McCabe and fixed by Colin Patrick McCabe (datanode , hdfs-client)
  The DataNode should use its shared memory segment to mark short-circuit replicas that have been unlinked as stale
  

• HDFS-6749. Major bug reported by Charles Lamb and fixed by Charles Lamb (namenode)
  FSNamesystem methods should call resolvePath
  

• HDFS-6740. Major sub-task reported by Lei (Eddy) Xu and fixed by Lei (Eddy) Xu (datanode)
  Make FSDataset support adding data volumes dynamically
  

• HDFS-6739. Major improvement reported by Tsz Wo Nicholas Sze and fixed by Tsz Wo Nicholas Sze (hdfs-client , namenode)
  Add getDatanodeStorageReport to ClientProtocol
  

• HDFS-6731. Major bug reported by WenJin Ma and fixed by Masatake Iwasaki (auto-failover , ha)
  Run "hdfs zkfc-formatZK" on a server in a non-namenode will cause a null pointer exception.
  

• HDFS-6728. Major sub-task reported by Lei (Eddy) Xu and fixed by Lei (Eddy) Xu (datanode)
  Dynamically add new volumes to DataStorage, formatted if necessary.
  

• HDFS-6727. Major sub-task reported by Lei (Eddy) Xu and fixed by Lei (Eddy) Xu (datanode)
  Refresh data volumes on DataNode based on configuration changes
  

• HDFS-6722. Major sub-task reported by Ming Ma and fixed by Ming Ma
  Display readable last contact time for dead nodes on NN webUI
  

• HDFS-6717. Minor sub-task reported by Jeff Hansen and fixed by Brandon Li (nfs)
  Jira HDFS-5804 breaks default nfs-gateway behavior for unsecured config
  

• HDFS-6715. Major bug reported by Arpit Gupta and fixed by Jing Zhao (ha , webhdfs)
  webhdfs wont fail over when it gets java.io.IOException: Namenode is in startup mode
  

• HDFS-6714. Minor bug reported by Vinayakumar B and fixed by Vinayakumar B (test)
  TestBlocksScheduledCounter#testBlocksScheduledCounter should shutdown cluster
  

• HDFS-6705. Major sub-task reported by Charles Lamb and fixed by Charles Lamb (namenode , security)
  Create an XAttr that disallows the HDFS admin from accessing a file
  

• HDFS-6704. Minor bug reported by Akira AJISAKA and fixed by Akira AJISAKA (documentation)
  Fix the command to launch JournalNode in HDFS-HA document
  

• HDFS-6702. Major bug reported by Tsz Wo Nicholas Sze and fixed by Tsz Wo Nicholas Sze (datanode , hdfs-client)
  DFSClient should create blocks using StorageType
  

• HDFS-6701. Major improvement reported by Ashwin Shankar and fixed by Ashwin Shankar (namenode)
  Make seed optional in NetworkTopology#sortByDistance
  

• HDFS-6700. Minor improvement reported by Tsz Wo Nicholas Sze and fixed by Tsz Wo Nicholas Sze (namenode)
  BlockPlacementPolicy shoud choose storage but not datanode for deletion
  

• HDFS-6693. Major bug reported by Vinayakumar B and fixed by Vinayakumar B (test , tools)
  TestDFSAdminWithHA fails on windows
  

• HDFS-6690. Major improvement reported by Andrew Wang and fixed by Andrew Wang (namenode)
  Deduplicate xattr names in memory
  

• HDFS-6689. Major bug reported by Yesha Vora and fixed by Brandon Li (nfs)
  NFS doesn't return correct lookup access for directories
  

• HDFS-6685. Major improvement reported by Tsz Wo Nicholas Sze and fixed by Tsz Wo Nicholas Sze (balancer & mover)
  Balancer should preserve storage type of replicas
  

• HDFS-6678. Minor bug reported by Chris Nauroth and fixed by Chris Nauroth (test)
  MiniDFSCluster may still be partially running after initialization fails.
  

• HDFS-6667. Major bug reported by Jian He and fixed by Jing Zhao (security)
  In HDFS HA mode, Distcp/SLive with webhdfs on secure cluster fails with Client cannot authenticate via:[TOKEN, KERBEROS] error
  

• HDFS-6665. Major test reported by Stephen Chu and fixed by Stephen Chu (hdfs-client)
  Add tests for XAttrs in combination with viewfs
  

• HDFS-6664. Trivial bug reported by Chris Nauroth and fixed by Ray Chiang (documentation)
  HDFS permissions guide documentation states incorrect default group mapping class.
  

• HDFS-6655. Major improvement reported by Vinayakumar B and fixed by Vinayakumar B
  Add 'header banner' to 'explorer.html' also in Namenode UI
  

• HDFS-6646. Major bug reported by Brahma Reddy Battula and fixed by Brahma Reddy Battula (tools)
  [ HDFS Rolling Upgrade - Shell ] shutdownDatanode and getDatanodeInfo usage is missed
  

• HDFS-6645. Minor test reported by Stephen Chu and fixed by Stephen Chu (snapshots , test)
  Add test for successive Snapshots between XAttr modifications
  

• HDFS-6643. Minor improvement reported by Tsz Wo Nicholas Sze and fixed by Tsz Wo Nicholas Sze (namenode)
  Refactor INodeFile.HeaderFormat and INodeWithAdditionalFields.PermissionStatusFormat
  

• HDFS-6640. Major bug reported by Brahma Reddy Battula and fixed by Stephen Chu (documentation , webhdfs)
  [ Web HDFS ] Syntax for MKDIRS, CREATESYMLINK, and SETXATTR are given wrongly(missed webhdfs/v1).).
  

• HDFS-6638. Major test reported by Liang Xie and fixed by Liang Xie (test)
  shorten test run time with a smaller retry timeout setting
  

• HDFS-6634. Major new feature reported by James Thomas and fixed by James Thomas (hdfs-client , namenode , qjm)
  inotify in HDFS
  

• HDFS-6630. Major bug reported by J.Andreina and fixed by Haohui Mai (namenode)
  Unable to fetch the block information by Browsing the file system on Namenode UI through IE9
  

• HDFS-6627. Major improvement reported by Liang Xie and fixed by Liang Xie (datanode)
  Rename DataNode#checkWriteAccess to checkReadAccess.
  

• HDFS-6621. Major bug reported by Benjamin Bowman and fixed by Rafal Wojdyla (balancer & mover)
  Hadoop Balancer prematurely exits iterations
  

• HDFS-6617. Minor test reported by Liang Xie and fixed by Liang Xie (auto-failover , test)
  Flake TestDFSZKFailoverController.testManualFailoverWithDFSHAAdmin due to a long edit log sync op
  

• HDFS-6616. Minor bug reported by zhaoyunjiong and fixed by zhaoyunjiong (webhdfs)
  bestNode shouldn't always return the first DataNode
  

• HDFS-6613. Minor improvement reported by Andrew Wang and fixed by Andrew Wang (caching)
  Improve logging in caching classes
  

• HDFS-6609. Major sub-task reported by Jing Zhao and fixed by Jing Zhao (namenode)
  Use DirectorySnapshottableFeature to represent a snapshottable directory
  

• HDFS-6606. Major improvement reported by Yi Liu and fixed by Yi Liu (datanode , hdfs-client , security)
  Optimize HDFS Encrypted Transport performance
  

  HDFS now supports the option to configure AES encryption for block data transfer. AES offers improved cryptographic strength and performance over the prior options of 3DES and RC4.

• HDFS-6597. Major improvement reported by Danilo Vunjak and fixed by Danilo Vunjak (namenode)
  Add a new option to NN upgrade to terminate the process after upgrade on NN is completed
  

• HDFS-6584. Major new feature reported by Tsz Wo Nicholas Sze and fixed by Tsz Wo Nicholas Sze (balancer & mover , namenode)
  Support Archival Storage
  

• HDFS-6582. Minor bug reported by Ted Yu and fixed by Abhiraj Butala (nfs)
  Missing null check in RpcProgramNfs3#read(XDR, SecurityHandler)
  

• HDFS-6581. Major new feature reported by Arpit Agarwal and fixed by Arpit Agarwal (datanode , hdfs-client , namenode)
  Write to single replica in memory
  

• HDFS-6570. Major improvement reported by Thejas M Nair and fixed by Jitendra Nath Pandey (hdfs-client , namenode , webhdfs)
  add api that enables checking if a user has certain permissions on a file
  

• HDFS-6569. Major bug reported by Brandon Li and fixed by Brandon Li (datanode)
  OOB message can't be sent to the client when DataNode shuts down for upgrade
  

• HDFS-6567. Major bug reported by Haohui Mai and fixed by Tassapol Athiapinya
  Normalize the order of public final in HdfsFileStatus
  

• HDFS-6544. Minor bug reported by Suraj Nayak M and fixed by Suraj Nayak M
  Broken Link for GFS in package.html
  

• HDFS-6534. Minor bug reported by Binglin Chang and fixed by Binglin Chang
  Fix build on macosx: HDFS parts
  

• HDFS-6519. Major improvement reported by Akira AJISAKA and fixed by Akira AJISAKA (documentation)
  Document oiv_legacy command
  

• HDFS-6517. Major bug reported by Akira AJISAKA and fixed by Akira AJISAKA
  Remove hadoop-metrics2.properties from hdfs project
  

• HDFS-6511. Minor improvement reported by Juan Yu and fixed by Juan Yu
  BlockManager#computeInvalidateWork() could do nothing
  

• HDFS-6506. Major bug reported by Binglin Chang and fixed by Binglin Chang (balancer & mover , test)
  Newly moved block replica been invalidated and deleted in TestBalancer
  

• HDFS-6482. Major improvement reported by James Thomas and fixed by James Thomas (datanode)
  Use block ID-based block layout on datanodes
  

  The directory structure for finalized replicas on DNs has been changed. Now, the directory that a finalized replica goes in is determined uniquely by its ID. Specifically, we use a two-level directory structure, with the 24th through 17th bits identifying the correct directory at the first level and the 16th through 8th bits identifying the correct directory at the second level.

• HDFS-6478. Major bug reported by Ming Ma and fixed by Ming Ma
  RemoteException can't be retried properly for non-HA scenario
  

• HDFS-6456. Major bug reported by Yesha Vora and fixed by Abhiraj Butala (nfs)
  NFS should throw error for invalid entry in dfs.nfs.exports.allowed.hosts
  

• HDFS-6455. Major bug reported by Yesha Vora and fixed by Abhiraj Butala (nfs)
  NFS: Exception should be added in NFS log for invalid separator in nfs.exports.allowed.hosts
  

• HDFS-6451. Major bug reported by Brandon Li and fixed by Abhiraj Butala (nfs)
  NFS should not return NFS3ERR_IO for AccessControlException
  

• HDFS-6441. Major improvement reported by Benoy Antony and fixed by Benoy Antony (balancer & mover)
  Add ability to exclude/include specific datanodes while balancing
  

• HDFS-6385. Major sub-task reported by Jing Zhao and fixed by Chris Nauroth
  Show when block deletion will start after NameNode startup in WebUI
  

• HDFS-6376. Major bug reported by Dave Marion and fixed by Dave Marion (datanode , federation , hdfs-client)
  Distcp data between two HA clusters requires another configuration
  

  Allow distcp to copy data between HA clusters. Users can use a new configuration property "dfs.internal.nameservices" to explicitly specify the name services belonging to the local cluster, while continue using the configuration property "dfs.nameservices" to specify all the name services in the local and remote clusters.

• HDFS-6247. Major bug reported by Vinayakumar B and fixed by Vinayakumar B (balancer & mover , datanode)
  Avoid timeouts for replaceBlock() call by sending intermediate responses to Balancer
  

• HDFS-6188. Major improvement reported by Benoy Antony and fixed by Benoy Antony (security)
  An ip whitelist based implementation of TrustedChannelResolver
  

• HDFS-6134. Major new feature reported by Alejandro Abdelnur and fixed by Charles Lamb (security)
  Transparent data at rest encryption
  

• HDFS-6114. Critical bug reported by Vinayakumar B and fixed by Vinayakumar B (datanode)
  Block Scan log rolling will never happen if blocks written continuously leading to huge size of dncp_block_verification.log.curr
  

• HDFS-6036. Major sub-task reported by Colin Patrick McCabe and fixed by Colin Patrick McCabe (caching , datanode)
  Forcibly timeout misbehaving DFSClients that try to do no-checksum reads that extend too long
  

• HDFS-5919. Major bug reported by Vinayakumar B and fixed by Vinayakumar B (namenode)
  FileJournalManager doesn't purge empty and corrupt inprogress edits files
  

• HDFS-5809. Critical bug reported by ikweesung and fixed by Colin Patrick McCabe (datanode)
  BlockPoolSliceScanner and high speed hdfs appending make datanode to drop into infinite loop
  

• HDFS-5723. Major bug reported by Vinayakumar B and fixed by Vinayakumar B (namenode)
  Append failed FINALIZED replica should not be accepted as valid when that block is underconstruction
  

• HDFS-5624. Major test reported by Chris Nauroth and fixed by Stephen Chu (hdfs-client , test)
  Add HDFS tests for ACLs in combination with viewfs.
  

• HDFS-5202. Major improvement reported by Colin Patrick McCabe and fixed by Chris Nauroth (datanode)
  Support Centralized Cache Management on Windows.
  

• HDFS-5185. Critical bug reported by Vinayakumar B and fixed by Vinayakumar B (datanode)
  DN fails to startup if one of the data dir is full
  

• HDFS-5182. Major improvement reported by Colin Patrick McCabe and fixed by Colin Patrick McCabe (hdfs-client)
  BlockReaderLocal must allow zero-copy reads only when the DN believes it's valid
  

• HDFS-5089. Major bug reported by Tsz Wo Nicholas Sze and fixed by Tsz Wo Nicholas Sze (namenode)
  When a LayoutVersion support SNAPSHOT, it must support FSIMAGE_NAME_OPTIMIZATION.
  

• HDFS-4852. Minor bug reported by Andrew Wang and fixed by Chris Nauroth
  libhdfs documentation is out of date
  

• HDFS-4629. Major bug reported by Amir Sanjar and fixed by (tools)
  Using com.sun.org.apache.xml.internal.serialize.* in XmlEditsVisitor.java is JVM vendor specific. Breaks IBM JAVA
  

• HDFS-4486. Minor improvement reported by Todd Lipcon and fixed by Zhe Zhang
  Add log category for long-running DFSClient notices
  

• HDFS-4257. Minor new feature reported by Harsh J and fixed by Tsz Wo Nicholas Sze (hdfs-client)
  The ReplaceDatanodeOnFailure policies could have a forgiving option
  

• HDFS-4227. Major bug reported by Eli Collins and fixed by Daisuke Kobayashi (documentation)
  Document dfs.namenode.resource.*
  

• HDFS-4165. Trivial bug reported by Binglin Chang and fixed by Binglin Chang (namenode)
  Faulty sanity check in FsDirectory.unprotectedSetQuota
  

• HDFS-4120. Minor improvement reported by Liang Xie and fixed by Rakesh R (ha , namenode)
  Add a new "-skipSharedEditsCheck" option for BootstrapStandby
  

• HDFS-3851. Trivial improvement reported by Jing Zhao and fixed by Jing Zhao (hdfs-client)
  Make DFSOuputSteram$Packet default constructor reuse the other constructor
  

• HDFS-3528. Major improvement reported by Todd Lipcon and fixed by James Thomas (datanode , hdfs-client , performance)
  Use native CRC32 in DFS write path
  

• HDFS-3482. Minor bug reported by Stephen Chu and fixed by madhukara phatak (balancer & mover)
  hdfs balancer throws ArrayIndexOutOfBoundsException if option is specified without arguments
  

• HDFS-2976. Trivial bug reported by Uma Maheswara Rao G and fixed by Uma Maheswara Rao G (hdfs-client)
  Remove unnecessary method (tokenRefetchNeeded) in DFSClient
  

• HDFS-2975. Major bug reported by Uma Maheswara Rao G and fixed by Yi Liu (namenode)
  Rename with overwrite flag true can make NameNode to stuck in safemode on NN (crash + restart).
  

• HDFS-2856. Major improvement reported by Owen O'Malley and fixed by Chris Nauroth (datanode , security)
  Fix block protocol so that Datanodes don't require root or jsvc
  

  SASL now can be used to secure the DataTransferProtocol, which transfers file block content between HDFS clients and DataNodes. In this configuration, it is no longer required for secured clusters to start the DataNode as root and bind to privileged ports.

• HDFS-573. Major improvement reported by Ziliang Guo and fixed by Chris Nauroth (libhdfs)
  Porting libhdfs to Windows
  

  The libhdfs C API is now supported on Windows.

• HADOOP-11286. Blocker bug reported by Christopher Tubbs and fixed by
  Map/Reduce dangerously adds Guava @Beta class to CryptoUtils
  

• HADOOP-11282. Trivial bug reported by Chris Nauroth and fixed by Chris Nauroth (test)
  Skip NFS TestShellBasedIdMapping tests that are irrelevant on Windows.
  

• HADOOP-11280. Trivial bug reported by Chris Nauroth and fixed by Chris Nauroth (native , test)
  TestWinUtils#testChmod fails after removal of NO_PROPAGATE_INHERIT_ACE.
  

• HADOOP-11265. Major bug reported by Larry McCay and fixed by Larry McCay (scripts)
  Credential and Key Shell Commands not available on Windows
  

• HADOOP-11260. Blocker bug reported by Karthik Kambatla and fixed by Mike Yoder (security)
  Patch up Jetty to disable SSLv3
  

• HADOOP-11254. Major bug reported by Zhijie Shen and fixed by Zhijie Shen
  Promoting AccessControlList to be public
  

• HADOOP-11253. Major test reported by Varun Vasudev and fixed by Varun Vasudev (tools)
  Hadoop streaming test TestStreamXmlMultipleRecords fails on Windows
  

• HADOOP-11250. Minor bug reported by Yi Liu and fixed by Yi Liu (build)
  fix endmacro of set_find_shared_library_without_version in CMakeLists
  

• HADOOP-11247. Major bug reported by Brandon Li and fixed by Brandon Li (nfs)
  Fix a couple javac warnings in NFS
  

• HADOOP-11243. Blocker bug reported by Wei Yan and fixed by Wei Yan
  SSLFactory shouldn't allow SSLv3
  

• HADOOP-11241. Major test reported by Varun Vasudev and fixed by Varun Vasudev
  TestNMSimulator fails sometimes due to timing issue
  

• HADOOP-11233. Minor bug reported by Steve Loughran and fixed by Stephen Chu (conf)
  hadoop.security.kms.client.encrypted.key.cache.expiry property spelled wrong in core-default
  

• HADOOP-11228. Major bug reported by Remus Rusanu and fixed by Remus Rusanu
  winutils task: unsecure path should not call AddNodeManagerAndUserACEsToObject
  

• HADOOP-11221. Major bug reported by Jinghui Wang and fixed by Jinghui Wang (util)
  JAVA specification for hashcode does not enforce it to be non-negative, but IdentityHashStore assumes System.identityHashCode() is non-negative
  

• HADOOP-11217. Blocker bug reported by Robert Kanter and fixed by Robert Kanter (kms)
  Disable SSLv3 in KMS
  

• HADOOP-11216. Major improvement reported by Yi Liu and fixed by Colin Patrick McCabe (security)
  Improve Openssl library finding
  

• HADOOP-11207. Major bug reported by Zhijie Shen and fixed by Zhijie Shen (security)
  DelegationTokenAuthenticationHandler needs to support DT operations for proxy user
  

• HADOOP-11198. Minor bug reported by Ted Yu and fixed by Li Lu
  Fix typo in javadoc for FileSystem#listStatus()
  

• HADOOP-11195. Major improvement reported by Yongjun Zhang and fixed by Yongjun Zhang (nfs , security)
  Move Id-Name mapping in NFS to the hadoop-common area for better maintenance
  

• HADOOP-11194. Major bug reported by Karthik Kambatla and fixed by Karthik Kambatla
  Ignore .keep files
  

• HADOOP-11193. Major bug reported by Xiaoyu Yao and fixed by Xiaoyu Yao (native)
  Fix uninitialized variables in NativeIO.c
  

• HADOOP-11184. Major improvement reported by Colin Patrick McCabe and fixed by Colin Patrick McCabe (native)
  Update Hadoop's lz4 to r123
  

• HADOOP-11182. Major bug reported by Sascha Coenen and fixed by Ravi Prakash
  GraphiteSink emits wrong timestamps
  

• HADOOP-11181. Major bug reported by Zhijie Shen and fixed by Zhijie Shen (security)
  o.a.h.security.token.delegation.DelegationTokenManager should be more generalized to handle other DelegationTokenIdentifier
  

• HADOOP-11179. Major bug reported by Hitesh Shah and fixed by Craig Welch
  Tarball as local resource type archive fails to localize on Windows
  

• HADOOP-11178. Minor bug reported by Arun Suresh and fixed by Arun Suresh (build)
  Fix findbugs exclude file
  

• HADOOP-11176. Major bug reported by Arun Suresh and fixed by Arun Suresh
  KMSClientProvider authentication fails when both currentUgi and loginUgi are a proxied user
  

• HADOOP-11175. Trivial bug reported by Yi Liu and fixed by Yi Liu (documentation , security)
  Fix several issues of hadoop security configuration in user doc.
  

• HADOOP-11174. Major bug reported by Yi Liu and fixed by Yi Liu (kms , security)
  Delegation token for KMS should only be got once if it already exists
  

• HADOOP-11170. Major bug reported by Arun Suresh and fixed by Arun Suresh
  ZKDelegationTokenSecretManager fails to renewToken created by a peer
  

• HADOOP-11169. Major bug reported by Arun Suresh and fixed by Arun Suresh
  Fix DelegationTokenAuthenticatedURL to pass the connection Configurator to the authenticator
  

• HADOOP-11168. Trivial bug reported by Yi Liu and fixed by Yi Liu (documentation)
  Remove duplicated entry "dfs.webhdfs.enabled" in the user doc
  

• HADOOP-11163. Minor bug reported by Chuan Liu and fixed by Chuan Liu (metrics)
  MetricsSystemImpl may miss a registered source
  

• HADOOP-11161. Major bug reported by Arun Suresh and fixed by Arun Suresh
  Expose close method in KeyProvider to give clients of Provider implementations a hook to release resources
  

• HADOOP-11160. Trivial bug reported by Charles Lamb and fixed by Charles Lamb (nfs)
  Fix typo in nfs3 server duplicate entry reporting
  

• HADOOP-11154. Trivial bug reported by Chris Nauroth and fixed by Chris Nauroth (documentation , native)
  Update BUILDING.txt to state that CMake 3.0 or newer is required on Mac.
  

• HADOOP-11153. Major improvement reported by Andrew Wang and fixed by Andrew Wang (kms)
  Make number of KMS threads configurable
  

• HADOOP-11151. Major bug reported by zhubin and fixed by Arun Suresh (security)
  Automatically refresh auth token and retry on auth failure
  

• HADOOP-11145. Major bug reported by Akira AJISAKA and fixed by Akira AJISAKA (test)
  TestFairCallQueue fails
  

• HADOOP-11143. Minor bug reported by Steve Loughran and fixed by Steve Loughran (net)
  NetUtils.wrapException loses inner stack trace on BindException
  

• HADOOP-11140. Major bug reported by Juan Yu and fixed by Juan Yu
  hadoop-aws only need test-scoped dependency on hadoop-common's tests jar
  

• HADOOP-11133. Minor bug reported by zhubin and fixed by Yi Liu (security)
  Should trim the content of keystore password file for JavaKeyStoreProvider
  

• HADOOP-11130. Major bug reported by Allen Wittenauer and fixed by Brandon Li (nfs)
  NFS updateMaps OS check is reversed
  

• HADOOP-11122. Blocker bug reported by Karthik Kambatla and fixed by Arun Suresh
  Fix findbugs in ZK DelegationTokenSecretManagers
  

• HADOOP-11113. Major bug reported by Arun Suresh and fixed by Arun Suresh (security)
  Namenode not able to reconnect to KMS after KMS restart
  

• HADOOP-11112. Major bug reported by Alejandro Abdelnur and fixed by Alejandro Abdelnur (security)
  TestKMSWithZK does not use KEY_PROVIDER_URI
  

• HADOOP-11111. Minor improvement reported by Steve Loughran and fixed by Steve Loughran (tools)
  MiniKDC to use locale EN_US for case conversions
  

• HADOOP-11110. Major bug reported by Andrew Wang and fixed by Arun Suresh
  JavaKeystoreProvider should not report a key as created if it was not flushed to the backing file
  

• HADOOP-11109. Major bug reported by Jian He and fixed by Jian He
  Site build is broken
  

• HADOOP-11106. Minor improvement reported by Andrew Wang and fixed by Charles Lamb (documentation)
  Document considerations of HAR and Encryption
  

• HADOOP-11105. Major bug reported by Chuan Liu and fixed by Chuan Liu (metrics)
  MetricsSystemImpl could leak memory in registered callbacks
  

• HADOOP-11101. Minor improvement reported by skrho and fixed by skrho
  How about inputstream close statement from catch block to finally block in FileContext#copy() ?
  

• HADOOP-11099. Major bug reported by Alejandro Abdelnur and fixed by Alejandro Abdelnur (security)
  KMS return HTTP UNAUTHORIZED 401 on ACL failure
  

• HADOOP-11097. Trivial bug reported by Charles Lamb and fixed by Charles Lamb (documentation)
  kms docs say proxyusers, not proxyuser for config params
  

• HADOOP-11096. Major bug reported by Alejandro Abdelnur and fixed by Alejandro Abdelnur (security)
  KMS: KeyAuthorizationKeyProvider should verify the keyversion belongs to the keyname on decrypt
  

• HADOOP-11091. Minor bug reported by David S. Wang and fixed by David S. Wang (fs/s3)
  Eliminate old configuration parameter names from s3a
  

• HADOOP-11088. Major test reported by Xiaoyu Yao and fixed by Xiaoyu Yao (security)
  Unittest TestKeyShell, TestCredShell and TestKMS assume UNIX path separator for JECKS key store path
  

• HADOOP-11085. Major bug reported by Mit Desai and fixed by Mit Desai
  Excessive logging by org.apache.hadoop.util.Progress when value is NaN
  

• HADOOP-11083. Major bug reported by Alejandro Abdelnur and fixed by Alejandro Abdelnur (security)
  After refactoring of HTTP proxyuser to common, doAs param is case sensitive
  

• HADOOP-11077. Major bug reported by Gregory Chanan and fixed by Gregory Chanan (security)
  NPE if hosts not specified in ProxyUsers
  

• HADOOP-11074. Major sub-task reported by David S. Wang and fixed by David S. Wang (fs/s3)
  Move s3-related FS connector code to hadoop-aws
  

• HADOOP-11073. Major test reported by Xiaoyu Yao and fixed by Xiaoyu Yao (security)
  Credential Provider related Unit Tests Failure on Windows
  

• HADOOP-11071. Minor test reported by Alejandro Abdelnur and fixed by Alejandro Abdelnur (security)
  KMSClientProvider should drain the local generated EEK cache on key rollover
  

• HADOOP-11070. Major test reported by Alejandro Abdelnur and fixed by Alejandro Abdelnur (security , test)
  Create MiniKMS for testing
  

• HADOOP-11069. Major bug reported by Alejandro Abdelnur and fixed by Alejandro Abdelnur (security)
  KMSClientProvider should use getAuthenticationMethod() to determine if in proxyuser mode or not
  

• HADOOP-11068. Major improvement reported by Gregory Chanan and fixed by Gregory Chanan (security)
  Match hadoop.auth cookie format to jetty output
  

• HADOOP-11067. Major bug reported by Yesha Vora and fixed by Xiaoyu Yao
  warning message 'ssl.client.truststore.location has not been set' gets printed for hftp command
  

• HADOOP-11064. Blocker bug reported by Steve Loughran and fixed by Chris Nauroth (native)
  UnsatisifedLinkError with hadoop 2.4 JARs on hadoop-2.6 due to NativeCRC32 method changes
  

• HADOOP-11063. Blocker bug reported by Chris Nauroth and fixed by Chris Nauroth
  KMS cannot deploy on Windows, because class names are too long.
  

• HADOOP-11062. Major bug reported by Alejandro Abdelnur and fixed by Arun Suresh (security , test)
  CryptoCodec testcases requiring OpenSSL should be run only if -Pnative is used
  

• HADOOP-11060. Major test reported by Alejandro Abdelnur and fixed by Yi Liu (security)
  Create a CryptoCodec test that verifies interoperability between the JCE and OpenSSL implementations
  

• HADOOP-11057. Minor improvement reported by Steve Loughran and fixed by Xiaoyu Yao (native)
  checknative command to probe for winutils.exe on windows
  

• HADOOP-11056. Major bug reported by Yongjun Zhang and fixed by Yongjun Zhang (security)
  OsSecureRandom.setConf() might leak file descriptors.
  

• HADOOP-11054. Major improvement reported by Alejandro Abdelnur and fixed by Alejandro Abdelnur (security)
  Add a KeyProvider instantiation based on a URI
  

• HADOOP-11049. Minor bug reported by Sangjin Lee and fixed by Sangjin Lee (util)
  javax package system class default is too broad
  

• HADOOP-11048. Minor bug reported by Sangjin Lee and fixed by Sangjin Lee (util)
  user/custom LogManager fails to load if the client classloader is enabled
  

• HADOOP-11040. Major bug reported by Yi Liu and fixed by Yi Liu (security)
  Return value of read(ByteBuffer buf) in CryptoInputStream is incorrect in some cases
  

• HADOOP-11036. Minor bug reported by Tsuyoshi OZAWA and fixed by Tsuyoshi OZAWA
  Add build directory to .gitignore
  

• HADOOP-11030. Minor improvement reported by Juan Yu and fixed by Juan Yu
  Define a variable jackson.version instead of using constant at multiple places
  

• HADOOP-11021. Minor improvement reported by Zhe Zhang and fixed by Zhe Zhang
  Configurable replication factor in the hadoop archive command
  

• HADOOP-11017. Major improvement reported by Alejandro Abdelnur and fixed by Arun Suresh (security)
  KMS delegation token secret manager should be able to use zookeeper as store
  

• HADOOP-11016. Major improvement reported by Alejandro Abdelnur and fixed by Alejandro Abdelnur (security)
  KMS should support signing cookies with zookeeper secret manager
  

• HADOOP-11015. Major improvement reported by Alejandro Abdelnur and fixed by Alejandro Abdelnur
  Http server/client utils to propagate and recreate Exceptions from server to client
  

• HADOOP-11012. Major bug reported by Eric Payne and fixed by Eric Payne (fs)
  hadoop fs -text of zero-length file causes EOFException
  

• HADOOP-11009. Major improvement reported by Gary Steelman and fixed by Gary Steelman (tools/distcp)
  Add Timestamp Preservation to DistCp
  

• HADOOP-11007. Major improvement reported by Jason Lowe and fixed by Jason Lowe (build , fs)
  Reinstate building of ant tasks support
  

• HADOOP-11005. Minor bug reported by Lei (Eddy) Xu and fixed by Lei (Eddy) Xu (conf)
  Fix HTTP content type for ReconfigurationServlet
  

• HADOOP-10998. Trivial improvement reported by Jim Hester and fixed by Jim Hester (scripts)
  Fix bash tab completion code to work
  

• HADOOP-10994. Major task reported by Alejandro Abdelnur and fixed by Alejandro Abdelnur (security)
  KeyProviderCryptoExtension should use CryptoCodec for generation/decryption of keys
  

• HADOOP-10992. Major task reported by Alejandro Abdelnur and fixed by Alejandro Abdelnur (security)
  Merge KMS to branch-2
  

• HADOOP-10990. Major improvement reported by Brandon Li and fixed by Brandon Li (nfs)
  Add missed NFSv3 request and response classes
  

• HADOOP-10989. Major bug reported by Chris Nauroth and fixed by Chris Nauroth (native)
  Work around buggy getgrouplist() implementations on Linux that return 0 on failure
  

• HADOOP-10982. Major improvement reported by Andrew Wang and fixed by Alejandro Abdelnur
  KMS: Support for multiple Kerberos principals
  

• HADOOP-10975. Major improvement reported by James Thomas and fixed by James Thomas (performance)
  org.apache.hadoop.util.DataChecksum should support native checksum calculation
  

• HADOOP-10973. Minor bug reported by Peter Klavins and fixed by Peter Klavins (documentation)
  Native Libraries Guide contains format error
  

• HADOOP-10972. Major bug reported by Peter Klavins and fixed by Peter Klavins (documentation)
  Native Libraries Guide contains mis-spelt build line
  

• HADOOP-10970. Major improvement reported by Andrew Wang and fixed by Andrew Wang
  Cleanup KMS configuration keys
  

• HADOOP-10968. Major bug reported by Dinar Valeev and fixed by (build)
  hadoop native build fails to detect java_libarch on ppc64le
  

• HADOOP-10967. Major improvement reported by Yi Liu and fixed by Yi Liu (security)
  Improve DefaultCryptoExtension#generateEncryptedKey performance
  

• HADOOP-10966. Blocker bug reported by Vinayakumar B and fixed by David Villegas (native)
  Hadoop Common native compilation broken in windows
  

• HADOOP-10964. Minor bug reported by Yi Liu and fixed by Yi Liu
  Small fix for NetworkTopologyWithNodeGroup#sortByDistance
  

• HADOOP-10962. Major bug reported by David Villegas and fixed by David Villegas (native)
  Flags for posix_fadvise are not valid in some architectures
  

• HADOOP-10954. Minor improvement reported by Masatake Iwasaki and fixed by Masatake Iwasaki (documentation)
  Adding site documents of hadoop-tools
  

• HADOOP-10946. Trivial bug reported by Ray Chiang and fixed by Ray Chiang
  Fix a bunch of typos in log messages
  

• HADOOP-10939. Major bug reported by Arun Suresh and fixed by Arun Suresh
  Fix TestKeyProviderFactory testcases to use default 128 bit length keys
  

• HADOOP-10937. Major bug reported by Arun Suresh and fixed by Arun Suresh (security)
  Need to set version name correctly before decrypting EEK
  

• HADOOP-10936. Major improvement reported by Andrew Wang and fixed by Andrew Wang
  Change default KeyProvider bitlength to 128
  

• HADOOP-10933. Major sub-task reported by Larry McCay and fixed by Larry McCay (security)
  FileBasedKeyStoresFactory Should use Configuration.getPassword for SSL Passwords
  

• HADOOP-10931. Minor bug reported by xukun and fixed by (build , fs/swift)
  compile error on project "Apache Hadoop OpenStack support"
  

• HADOOP-10929. Trivial bug reported by Larry McCay and fixed by Larry McCay (security)
  Typo in Configuration.getPasswordFromCredentialProviders
  

• HADOOP-10928. Trivial bug reported by Josh Elser and fixed by Josh Elser (security)
  Incorrect usage on 'hadoop credential list'
  

• HADOOP-10927. Minor bug reported by Josh Elser and fixed by Josh Elser (security)
  Fix CredentialShell help behavior and error codes
  

• HADOOP-10925. Blocker bug reported by Chris Nauroth and fixed by Chris Nauroth (native)
  Compilation fails in native link0 function on Windows.
  

• HADOOP-10922. Major improvement reported by Andrew Wang and fixed by Larry McCay
  User documentation for CredentialShell
  

• HADOOP-10920. Minor bug reported by Ted Yu and fixed by Akira AJISAKA (documentation)
  site plugin couldn't parse hadoop-kms index.apt.vm
  

• HADOOP-10918. Major bug reported by Alejandro Abdelnur and fixed by Alejandro Abdelnur
  JMXJsonServlet fails when used within Tomcat
  

• HADOOP-10911. Major bug reported by Gregory Chanan and fixed by (security)
  hadoop.auth cookie after HADOOP-10710 still not proper according to RFC2109
  

• HADOOP-10905. Major sub-task reported by Larry McCay and fixed by Larry McCay (security)
  LdapGroupsMapping Should use configuration.getPassword for SSL and LDAP Passwords
  

• HADOOP-10903. Major improvement reported by Chris Nauroth and fixed by Chris Nauroth (scripts , util)
  Enhance hadoop classpath command to expand wildcards or write classpath into jar manifest.
  

  The "hadoop classpath" command has been enhanced to support options for automatic expansion of wildcards in classpath elements and writing the classpath to a jar file manifest. These options make it easier to construct a correct classpath for libhdfs applications.

• HADOOP-10902. Minor improvement reported by Stephen Chu and fixed by Stephen Chu
  Deletion of directories with snapshots will not output reason for trash move failure
  

• HADOOP-10900. Minor bug reported by Andrew Wang and fixed by Andrew Wang
  CredentialShell args should use single-dash style
  

• HADOOP-10893. Major new feature reported by Sangjin Lee and fixed by Sangjin Lee (util)
  isolated classloader on the client side
  

• HADOOP-10891. Major improvement reported by Andrew Wang and fixed by Andrew Wang
  Add EncryptedKeyVersion factory method to KeyProviderCryptoExtension
  

• HADOOP-10887. Major bug reported by Stephen Chu and fixed by Stephen Chu (fs , test)
  Add XAttrs to ViewFs and make XAttrs + ViewFileSystem internal dir behavior consistent
  

• HADOOP-10886. Major sub-task reported by Uma Maheswara Rao G and fixed by Uma Maheswara Rao G (fs)
  CryptoCodec#getCodecclasses throws NPE when configurations not loaded.
  

• HADOOP-10884. Minor sub-task reported by Akira AJISAKA and fixed by Akira AJISAKA (documentation)
  Fix dead link in Configuration javadoc
  

• HADOOP-10882. Minor task reported by Todd Lipcon and fixed by Todd Lipcon (util)
  Move DirectBufferPool into common util
  

• HADOOP-10881. Major improvement reported by Andrew Wang and fixed by Andrew Wang
  Clarify usage of encryption and encrypted encryption key in KeyProviderCryptoExtension
  

• HADOOP-10880. Blocker bug reported by Alejandro Abdelnur and fixed by Alejandro Abdelnur (security)
  Move HTTP delegation tokens out of URL querystring to a header
  

• HADOOP-10868. Major sub-task reported by Robert Kanter and fixed by Robert Kanter (security)
  Create a ZooKeeper-backed secret provider
  

• HADOOP-10866. Major bug reported by Yongjun Zhang and fixed by Yongjun Zhang
  RawLocalFileSystem fails to read symlink targets via the stat command when the format of stat command uses non-curly quotes
  

• HADOOP-10863. Major improvement reported by Alejandro Abdelnur and fixed by Arun Suresh (security)
  KMS should have a blacklist for decrypting EEKs
  

• HADOOP-10862. Major bug reported by Alejandro Abdelnur and fixed by Arun Suresh (security)
  Miscellaneous trivial corrections to KMS classes
  

• HADOOP-10857. Major bug reported by Tsuyoshi OZAWA and fixed by Tsuyoshi OZAWA (documentation)
  Native Libraries Guide doen't mention a dependency on openssl-development package
  

• HADOOP-10855. Minor improvement reported by Todd Lipcon and fixed by Todd Lipcon (io)
  Allow Text to be read with a known length
  

• HADOOP-10851. Major bug reported by Benoy Antony and fixed by Benoy Antony (security)
  NetgroupCache does not remove group memberships
  

• HADOOP-10845. Major improvement reported by Chris Nauroth and fixed by Stephen Chu (fs , test)
  Add common tests for ACLs in combination with viewfs.
  

• HADOOP-10843. Major bug reported by Jinghui Wang and fixed by Jinghui Wang (test , tools)
  TestGridmixRecord unit tests failure on PowerPC
  

• HADOOP-10842. Major bug reported by Alejandro Abdelnur and fixed by Arun Suresh (security)
  CryptoExtension generateEncryptedKey method should receive the key name
  

• HADOOP-10841. Major improvement reported by Alejandro Abdelnur and fixed by Arun Suresh (security)
  EncryptedKeyVersion should have a key name property
  

• HADOOP-10839. Major improvement reported by shanyu zhao and fixed by shanyu zhao (metrics)
  Add unregisterSource() to MetricsSystem API
  

  The MetricsSystem abstract class has added a new abstract method, unregisterSource, for unregistering a previously registered metrics source. Custom subclasses of MetricsSystem must be updated to provide an implementation of this method.

• HADOOP-10838. Major improvement reported by James Thomas and fixed by James Thomas (performance)
  Byte array native checksumming
  

• HADOOP-10835. Major improvement reported by Alejandro Abdelnur and fixed by Alejandro Abdelnur (security)
  Implement HTTP proxyuser support in HTTP authentication client/server libraries
  

• HADOOP-10833. Major improvement reported by Benoy Antony and fixed by Benoy Antony (security)
  Remove unused cache in UserProvider
  

• HADOOP-10830. Major bug reported by Benoy Antony and fixed by Benoy Antony (security)
  Missing lock in JavaKeyStoreProvider.createCredentialEntry
  

• HADOOP-10826. Major improvement reported by Benoy Antony and fixed by Benoy Antony (security)
  Iteration on KeyProviderFactory.serviceLoader is thread-unsafe
  

• HADOOP-10824. Major improvement reported by Benoy Antony and fixed by Benoy Antony (security)
  Refactor KMSACLs to avoid locking
  

• HADOOP-10820. Minor bug reported by Alex Holmes and fixed by zhihai xu
  Throw an exception in GenericOptionsParser when passed an empty Path
  

• HADOOP-10817. Major improvement reported by Alejandro Abdelnur and fixed by Alejandro Abdelnur (security)
  ProxyUsers configuration should support configurable prefixes
  

• HADOOP-10816. Major bug reported by Mike Yoder and fixed by Mike Yoder (security)
  KeyShell returns -1 on error to the shell, should be 1
  

• HADOOP-10815. Major improvement reported by Chris Nauroth and fixed by Chris Nauroth (native)
  Implement Windows equivalent of mlock.
  

• HADOOP-10814. Major bug reported by Alejandro Abdelnur and fixed by Robert Kanter
  Update Tomcat version used by HttpFS and KMS to latest 6.x version
  

• HADOOP-10812. Trivial improvement reported by Andrew Wang and fixed by Andrew Wang
  Delegate KeyProviderExtension#toString to underlying KeyProvider
  

• HADOOP-10810. Minor bug reported by Chris Nauroth and fixed by Chris Nauroth (native)
  Clean up native code compilation warnings.
  

• HADOOP-10808. Minor improvement reported by Chris Nauroth and fixed by Chris Nauroth (native)
  Remove unused native code for munlock.
  

• HADOOP-10793. Major improvement reported by Mike Yoder and fixed by Andrew Wang (security)
  KeyShell args should use single-dash style
  

• HADOOP-10791. Major improvement reported by Alejandro Abdelnur and fixed by Robert Kanter (security)
  AuthenticationFilter should support externalizing the secret for signing and provide rotation support
  

• HADOOP-10781. Major bug reported by Dmitry Sivachenko and fixed by Dmitry Sivachenko
  Unportable getgrouplist() usage breaks FreeBSD
  

• HADOOP-10780. Major bug reported by Dmitry Sivachenko and fixed by Dmitry Sivachenko
  hadoop_user_info_alloc fails on FreeBSD due to incorrect sysconf use
  

• HADOOP-10771. Major improvement reported by Alejandro Abdelnur and fixed by Alejandro Abdelnur (security)
  Refactor HTTP delegation support out of httpfs to common
  

• HADOOP-10770. Major improvement reported by Alejandro Abdelnur and fixed by Alejandro Abdelnur (security)
  KMS add delegation token support
  

• HADOOP-10769. Major improvement reported by Alejandro Abdelnur and fixed by Arun Suresh (security)
  Create KeyProvider extension to handle delegation tokens
  

• HADOOP-10758. Major improvement reported by Alejandro Abdelnur and fixed by Arun Suresh (security)
  KMS: add ACLs on per key basis.
  

• HADOOP-10757. Major improvement reported by Alejandro Abdelnur and fixed by Arun Suresh (security)
  KeyProvider KeyVersion should provide the key name
  

• HADOOP-10756. Major improvement reported by Alejandro Abdelnur and fixed by Arun Suresh (security)
  KMS audit log should consolidate successful similar requests
  

• HADOOP-10755. Major improvement reported by Andrew Wang and fixed by Lei (Eddy) Xu (security)
  Support negative caching of user-group mapping
  

• HADOOP-10750. Major improvement reported by Alejandro Abdelnur and fixed by Arun Suresh (security)
  KMSKeyProviderCache should be in hadoop-common
  

• HADOOP-10736. Major improvement reported by Mike Yoder and fixed by Mike Yoder (security)
  Add key attributes to the key shell
  

• HADOOP-10733. Minor bug reported by Ted Yu and fixed by Ted Yu
  Potential null dereference in CredentialShell#promptForCredential()
  

• HADOOP-10732. Minor bug reported by Ted Yu and fixed by Ted Yu
  Update without holding write lock in JavaKeyStoreProvider#innerSetCredential()
  

• HADOOP-10731. Trivial improvement reported by Henry Saputra and fixed by Henry Saputra (documentation)
  Remove @date JavaDoc comment in ProgramDriver class
  

• HADOOP-10720. Major improvement reported by Alejandro Abdelnur and fixed by Arun Suresh (security)
  KMS: Implement generateEncryptedKey and decryptEncryptedKey in the REST API
  

• HADOOP-10719. Major new feature reported by Alejandro Abdelnur and fixed by Arun Suresh (security)
  Add generateEncryptedKey and decryptEncryptedKey methods to KeyProvider
  

• HADOOP-10698. Major improvement reported by Alejandro Abdelnur and fixed by Alejandro Abdelnur (security)
  KMS, add proxyuser support
  

• HADOOP-10696. Major improvement reported by Alejandro Abdelnur and fixed by Alejandro Abdelnur (security)
  Add optional attributes to KeyProvider Options and Metadata
  

• HADOOP-10695. Major improvement reported by Andrew Wang and fixed by Mike Yoder
  KMSClientProvider should respect a configurable timeout.
  

• HADOOP-10681. Major bug reported by Gopal V and fixed by Gopal V (performance)
  Remove synchronized blocks from SnappyCodec and ZlibCodec buffering inner loop
  

  Remove unnecessary synchronized blocks from Snappy/Zlib codecs.

• HADOOP-10677. Major bug reported by David S. Wang and fixed by David S. Wang (fs/s3)
  ExportSnapshot fails on kerberized cluster using s3a
  

• HADOOP-10676. Major bug reported by David S. Wang and fixed by David S. Wang (fs/s3)
  S3AOutputStream not reading new config knobs for multipart configs
  

• HADOOP-10675. Major improvement reported by David S. Wang and fixed by David S. Wang (fs/s3)
  Add server-side encryption functionality to s3a
  

• HADOOP-10673. Major bug reported by Ming Ma and fixed by Ming Ma
  Update rpc metrics when the call throws an exception
  

• HADOOP-10650. Major sub-task reported by Benoy Antony and fixed by Benoy Antony (security)
  Add ability to specify a reverse ACL (black list) of users and groups
  

• HADOOP-10645. Major bug reported by Alejandro Abdelnur and fixed by Alejandro Abdelnur (security)
  TestKMS fails because race condition writing acl files
  

• HADOOP-10635. Major sub-task reported by Alejandro Abdelnur and fixed by Yi Liu (security)
  Add a method to CryptoCodec to generate SRNs for IV
  

• HADOOP-10632. Major sub-task reported by Alejandro Abdelnur and fixed by Yi Liu (security)
  Minor improvements to Crypto input and output streams
  

• HADOOP-10611. Major bug reported by Alejandro Abdelnur and fixed by Alejandro Abdelnur (security)
  KMS, keyVersion name should not be assumed to be keyName@versionNumber
  

• HADOOP-10610. Minor improvement reported by Ted Malaska and fixed by Ted Malaska (fs/s3)
  Upgrade S3n s3.fs.buffer.dir to support multi directories
  

• HADOOP-10607. Major new feature reported by Larry McCay and fixed by Larry McCay (security)
  Create an API to Separate Credentials/Password Storage from Applications
  

• HADOOP-10591. Major bug reported by Hari Shreedharan and fixed by Colin Patrick McCabe
  Compression codecs must used pooled direct buffers or deallocate direct buffers when stream is closed
  

• HADOOP-10586. Minor bug reported by Charles Lamb and fixed by Charles Lamb (bin)
  KeyShell doesn't allow setting Options via CLI
  

• HADOOP-10583. Minor bug reported by Charles Lamb and fixed by Charles Lamb (bin)
  bin/hadoop key throws NPE with no args and assorted other fixups
  

  bin/hadoop key with no args would throw an NPE.

• HADOOP-10552. Trivial bug reported by Kenji Kikushima and fixed by Kenji Kikushima (documentation)
  Fix usage and example at FileSystemShell.apt.vm
  

• HADOOP-10534. Major bug reported by Owen O'Malley and fixed by Owen O'Malley
  KeyProvider API should using windowing for retrieving metadata
  

• HADOOP-10507. Minor bug reported by Stephen Chu and fixed by sathish (fs)
  FsShell setfacl can throw ArrayIndexOutOfBoundsException when no perm is specified
  

• HADOOP-10488. Major bug reported by Alejandro Abdelnur and fixed by Alejandro Abdelnur (test)
  TestKeyProviderFactory fails randomly
  

• HADOOP-10433. Major improvement reported by Alejandro Abdelnur and fixed by Alejandro Abdelnur (security)
  Key Management Server based on KeyProvider API
  

• HADOOP-10432. Major improvement reported by Alejandro Abdelnur and fixed by Alejandro Abdelnur (security)
  Refactor SSLFactory to expose static method to determine HostnameVerifier
  

• HADOOP-10431. Major improvement reported by Alejandro Abdelnur and fixed by Alejandro Abdelnur (security)
  Change visibility of KeyStore.Options getter methods to public
  

• HADOOP-10430. Major improvement reported by Alejandro Abdelnur and fixed by Alejandro Abdelnur (security)
  KeyProvider Metadata should have an optional description, there should be a method to retrieve the metadata from all keys
  

• HADOOP-10429. Major improvement reported by Alejandro Abdelnur and fixed by Alejandro Abdelnur (security)
  KeyStores should have methods to generate the materials themselves, KeyShell should use them
  

• HADOOP-10428. Major improvement reported by Alejandro Abdelnur and fixed by Alejandro Abdelnur (security)
  JavaKeyStoreProvider should accept keystore password via configuration falling back to ENV VAR
  

• HADOOP-10427. Major improvement reported by Alejandro Abdelnur and fixed by Alejandro Abdelnur (security)
  KeyProvider implementations should be thread safe
  

• HADOOP-10404. Minor bug reported by Ted Yu and fixed by Colin Patrick McCabe
  Some accesses to DomainSocketWatcher#closed are not protected by lock
  

• HADOOP-10402. Major bug reported by Robert Kanter and fixed by Robert Kanter
  Configuration.getValByRegex does not substitute for variables
  

• HADOOP-10400. Major new feature reported by Jordan Mendelson and fixed by Jordan Mendelson (fs , fs/s3)
  Incorporate new S3A FileSystem implementation
  

• HADOOP-10373. Major improvement reported by Steve Loughran and fixed by Steve Loughran (fs/s3)
  create tools/hadoop-amazon for aws/EMR support
  

• HADOOP-10335. Major improvement reported by Benoy Antony and fixed by Benoy Antony
  An ip whilelist based implementation to resolve Sasl properties per connection
  

• HADOOP-10281. Major sub-task reported by Chris Li and fixed by Chris Li
  Create a scheduler, which assigns schedulables a priority level
  

• HADOOP-10244. Major bug reported by Larry McCay and fixed by Larry McCay (security)
  TestKeyShell improperly tests the results of a Delete
  

  Fix of inappropriate test of delete functionality.

• HADOOP-10237. Major bug reported by Larry McCay and fixed by Larry McCay (security)
  JavaKeyStoreProvider needs to set keystore permissions properly
  

• HADOOP-10231. Minor improvement reported by Akira AJISAKA and fixed by Akira AJISAKA (documentation)
  Add some components in Native Libraries document
  

• HADOOP-10224. Major bug reported by Larry McCay and fixed by Arun Suresh (security)
  JavaKeyStoreProvider has to protect against corrupting underlying store
  

• HADOOP-10201. Major sub-task reported by Larry McCay and fixed by Larry McCay (security)
  Add Listing Support to Key Management APIs
  

  I just committed this. Thanks, Larry!

• HADOOP-10177. Major bug reported by Owen O'Malley and fixed by Larry McCay (security)
  Create CLI tools for managing keys via the KeyProvider API
  

• HADOOP-10150. Major new feature reported by Yi Liu and fixed by Yi Liu (security)
  Hadoop cryptographic file system
  

• HADOOP-10141. Major bug reported by Owen O'Malley and fixed by Owen O'Malley (security)
  Create an API to separate encryption key storage from applications
  

• HADOOP-10131. Major bug reported by Vinayakumar B and fixed by Vinayakumar B
  NetWorkTopology#countNumOfAvailableNodes() is returning wrong value if excluded nodes passed are not part of the cluster tree
  

• HADOOP-10121. Trivial bug reported by Akira AJISAKA and fixed by Akira AJISAKA (documentation)
  Fix javadoc spelling for HadoopArchives#writeTopLevelDirs
  

• HADOOP-10059. Minor bug reported by Jason Lowe and fixed by Tsuyoshi OZAWA (metrics)
  RPC authentication and authorization metrics overflow to negative values on busy clusters
  

• HADOOP-9989. Major bug reported by Jinghui Wang and fixed by zhihai xu (security , util)
  Bug introduced in HADOOP-9374, which parses the -tokenCacheFile as binary file but set it to the configuration as JSON file.
  

• HADOOP-9921. Major bug reported by Vinayakumar B and fixed by Vinayakumar B
  daemon scripts should remove pid file on stop call after stop or process is found not running
  

• HADOOP-9740. Major bug reported by Allan Yan and fixed by Allan Yan (fs)
  FsShell's Text command does not read avro data files stored on HDFS
  

• HADOOP-9576. Major bug reported by Jian He and fixed by Steve Loughran
  Make NetUtils.wrapException throw EOFException instead of wrapping it as IOException
  

• HADOOP-9540. Minor improvement reported by Hari and fixed by (fs/s3 , test)
  Expose the InMemoryS3 and S3N FilesystemStores implementations for Unit testing.
  

• HADOOP-8944. Trivial improvement reported by Jonathan Allen and fixed by Jonathan Allen
  Shell command fs -count should include human readable option
  

  Implements -h option for fs -count to show file sizes in human readable format. Additionally, ContentSummary.getHeader() now returns a different string that is incompatible with previous releases.

• HADOOP-8896. Trivial improvement reported by Timothy Mann and fixed by Ray Chiang (documentation , io)
  Javadoc points to Wrong Reader and Writer classes in SequenceFile
  

• HADOOP-8815. Minor improvement reported by Brandon Li and fixed by Brandon Li (test)
  RandomDatum overrides equals(Object) but no hashCode()
  

• HADOOP-8808. Major bug reported by Hemanth Yamijala and fixed by Akira AJISAKA (documentation , fs)
  Update FsShell documentation to mention deprecation of some of the commands, and mention alternatives
  

• HADOOP-8158. Major bug reported by Todd Lipcon and fixed by Daryn Sharp
  Interrupting hadoop fs -put from the command line causes a LeaseExpiredException
  

• HADOOP-8069. Major improvement reported by Todd Lipcon and fixed by Todd Lipcon (ipc)
  Enable TCP_NODELAY by default for IPC
  

  This change enables the TCP_NODELAY flag for all Hadoop IPC connections, hence bypassing TCP Nagling. Nagling interacts poorly with TCP delayed ACKs especially for request-response protocols.

• HADOOP-7664. Minor improvement reported by Ravi Prakash and fixed by Ravi Prakash (conf)
  o.a.h.conf.Configuration complains of overriding final parameter even if the value with which its attempting to override is the same.
  

Hadoop 2.4.1 Release Notes

• YARN-2081. Minor bug reported by Hong Zhiguo and fixed by Hong Zhiguo (applications/distributed-shell)
  TestDistributedShell fails after YARN-1962
  

  java.lang.AssertionError: expected:<1> but was:<0> at org.junit.Assert.fail(Assert.java:88) at org.junit.Assert.failNotEquals(Assert.java:743) at org.junit.Assert.assertEquals(Assert.java:118) at org.junit.Assert.assertEquals(Assert.java:555) at org.junit.Assert.assertEquals(Assert.java:542) at org.apache.hadoop.yarn.applications.distributedshell.TestDistributedShell.testDSShell(TestDistributedShell.java:198)

• YARN-2066. Minor bug reported by Ted Yu and fixed by Hong Zhiguo
  Wrong field is referenced in GetApplicationsRequestPBImpl#mergeLocalToBuilder()
  

  {code} if (this.finish != null) { builder.setFinishBegin(start.getMinimumLong()); builder.setFinishEnd(start.getMaximumLong()); } {code} this.finish should be referenced in the if block.

• YARN-2053. Major sub-task reported by Sumit Mohanty and fixed by Wangda Tan (resourcemanager)
  Slider AM fails to restart: NPE in RegisterApplicationMasterResponseProto$Builder.addAllNmTokensFromPreviousAttempts
  

  Slider AppMaster restart fails with the following: {code} org.apache.hadoop.yarn.proto.YarnServiceProtos$RegisterApplicationMasterResponseProto$Builder.addAllNmTokensFromPreviousAttempts(YarnServiceProtos.java:2700) {code}

• YARN-2016. Major bug reported by Venkat Ranganathan and fixed by Junping Du (resourcemanager)
  Yarn getApplicationRequest start time range is not honored
  

  When we query for the previous applications by creating an instance of GetApplicationsRequest and setting the start time range and application tag, we see that the start range provided is not honored and all applications with the tag are returned Attaching a reproducer.

• YARN-1986. Critical bug reported by Jon Bringhurst and fixed by Hong Zhiguo
  In Fifo Scheduler, node heartbeat in between creating app and attempt causes NPE
  

  After upgrade from 2.2.0 to 2.4.0, NPE on first job start. -After RM was restarted, the job runs without a problem.- {noformat} 19:11:13,441 FATAL ResourceManager:600 - Error in handling event type NODE_UPDATE to the scheduler java.lang.NullPointerException at org.apache.hadoop.yarn.server.resourcemanager.scheduler.fifo.FifoScheduler.assignContainers(FifoScheduler.java:462) at org.apache.hadoop.yarn.server.resourcemanager.scheduler.fifo.FifoScheduler.nodeUpdate(FifoScheduler.java:714) at org.apache.hadoop.yarn.server.resourcemanager.scheduler.fifo.FifoScheduler.handle(FifoScheduler.java:743) at org.apache.hadoop.yarn.server.resourcemanager.scheduler.fifo.FifoScheduler.handle(FifoScheduler.java:104) at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$SchedulerEventDispatcher$EventProcessor.run(ResourceManager.java:591) at java.lang.Thread.run(Thread.java:744) 19:11:13,443 INFO ResourceManager:604 - Exiting, bbye.. {noformat}

• YARN-1976. Major bug reported by Yesha Vora and fixed by Junping Du
  Tracking url missing http protocol for FAILED application
  

  Run yarn application -list -appStates FAILED, It does not print http protocol name like FINISHED apps. {noformat} -bash-4.1$ yarn application -list -appStates FINISHED,FAILED,KILLED 14/04/15 23:55:07 INFO client.RMProxy: Connecting to ResourceManager at host Total number of applications (application-types: [] and states: [FINISHED, FAILED, KILLED]):4 Application-Id Application-Name Application-Type User Queue State Final-State Progress Tracking-URL application_1397598467870_0004 Sleep job MAPREDUCE hrt_qa default FINISHED SUCCEEDED 100% http://host:19888/jobhistory/job/job_1397598467870_0004 application_1397598467870_0003 Sleep job MAPREDUCE hrt_qa default FINISHED SUCCEEDED 100% http://host:19888/jobhistory/job/job_1397598467870_0003 application_1397598467870_0002 Sleep job MAPREDUCE hrt_qa default FAILED FAILED 100% host:8088/cluster/app/application_1397598467870_0002 application_1397598467870_0001 word count MAPREDUCE hrt_qa default FINISHED SUCCEEDED 100% http://host:19888/jobhistory/job/job_1397598467870_0001 {noformat} It only prints 'host:8088/cluster/app/application_1397598467870_0002' instead 'http://host:8088/cluster/app/application_1397598467870_0002'

• YARN-1975. Major bug reported by Nathan Roberts and fixed by Mit Desai (resourcemanager)
  Used resources shows escaped html in CapacityScheduler and FairScheduler page
  

  Used resources displays as &amp;lt;memory:1111, vCores;&amp;gt; with capacity scheduler

• YARN-1962. Major sub-task reported by Mohammad Kamrul Islam and fixed by Mohammad Kamrul Islam
  Timeline server is enabled by default
  

  Since Timeline server is not matured and secured yet, enabling it by default might create some confusion. We were playing with 2.4.0 and found a lot of exceptions for distributed shell example related to connection refused error. Btw, we didn't run TS because it is not secured yet. Although it is possible to explicitly turn it off through yarn-site config. In my opinion, this extra change for this new service is not worthy at this point,. This JIRA is to turn it off by default. If there is an agreement, i can put a simple patch about this. {noformat} 14/04/17 23:24:33 ERROR impl.TimelineClientImpl: Failed to get the response from the timeline server. com.sun.jersey.api.client.ClientHandlerException: java.net.ConnectException: Connection refused at com.sun.jersey.client.urlconnection.URLConnectionClientHandler.handle(URLConnectionClientHandler.java:149) at com.sun.jersey.api.client.Client.handle(Client.java:648) at com.sun.jersey.api.client.WebResource.handle(WebResource.java:670) at com.sun.jersey.api.client.WebResource.access$200(WebResource.java:74) at com.sun.jersey.api.client.WebResource$Builder.post(WebResource.java:563) at org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl.doPostingEntities(TimelineClientImpl.java:131) at org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl.putEntities(TimelineClientImpl.java:104) at org.apache.hadoop.yarn.applications.distributedshell.ApplicationMaster.publishApplicationAttemptEvent(ApplicationMaster.java:1072) at org.apache.hadoop.yarn.applications.distributedshell.ApplicationMaster.run(ApplicationMaster.java:515) at org.apache.hadoop.yarn.applications.distributedshell.ApplicationMaster.main(ApplicationMaster.java:281) Caused by: java.net.ConnectException: Connection refused at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:198) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:579) at java.net.Socket.connect(Socket.java:528) at sun.net.NetworkClient.doConnect(NetworkClient.java:180) at sun.net.www.http.HttpClient.openServer(HttpClient.java:432) at sun.net.www.http.HttpClient.openServer(HttpClient.java:527) at sun.net.www.http.HttpClient.<in14/04/17 23:24:33 ERROR impl.TimelineClientImpl: Failed to get the response from the timeline server. com.sun.jersey.api.client.ClientHandlerException: java.net.ConnectException: Connection refused at com.sun.jersey.client.urlconnection.URLConnectionClientHandler.handle(URLConnectionClientHandler.java:149) at com.sun.jersey.api.client.Client.handle(Client.java:648) at com.sun.jersey.api.client.WebResource.handle(WebResource.java:670) at com.sun.jersey.api.client.WebResource.access$200(WebResource.java:74) at com.sun.jersey.api.client.WebResource$Builder.post(WebResource.java:563) at org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl.doPostingEntities(TimelineClientImpl.java:131) at org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl.putEntities(TimelineClientImpl.java:104) at org.apache.hadoop.yarn.applications.distributedshell.ApplicationMaster.publishApplicationAttemptEvent(ApplicationMaster.java:1072) at org.apache.hadoop.yarn.applications.distributedshell.ApplicationMaster.run(ApplicationMaster.java:515) at org.apache.hadoop.yarn.applications.distributedshell.ApplicationMaster.main(ApplicationMaster.java:281) Caused by: java.net.ConnectException: Connection refused at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:198) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:579) at java.net.Socket.connect(Socket.java:528) at sun.net.NetworkClient.doConnect(NetworkClient.java:180) at sun.net.www.http.HttpClient.openServer(HttpClient.java:432) at sun.net.www.http.HttpClient.openServer(HttpClient.java:527) at sun.net.www.http.HttpClient.<init>(HttpClient.java:211) at sun.net.www.http.HttpClient.New(HttpClient.java:308) at sun.net.www.http.HttpClient.New(HttpClient.java:326) at sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:996) at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:932) at sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:850) at sun.net.www.protocol.http.HttpURLConnection.getOutputStream(HttpURLConnection.java:1091) at com.sun.jersey.client.urlconnection.URLConnectionClientHandler$1$1.getOutputStream(URLConnectionClientHandler.java:225) at com.sun.jersey.api.client.CommittingOutputStream.commitWrite(CommittingOutputStream.java:117) at com.sun.jersey.api.client.CommittingOutputStream.write(CommittingOutputStream.java:89) at org.codehaus.jackson.impl.Utf8Generator._flushBuffer(Utf8Generator.java:1754) at org.codehaus.jackson.impl.Utf8Generator.flush(Utf8Generator.java:1088) at org.codehaus.jackson.map.ObjectMapper.writeValue(ObjectMapper.java:1354) at org.codehaus.jackson.jaxrs.JacksonJsonProvider.writeTo(JacksonJsonProvider.java:527) at com.sun.jersey.api.client.RequestWriter.writeRequestEntity(RequestWriter.java:300) at com.sun.jersey.client.urlconnection.URLConnectionClientHandler._invoke(URLConnectionClientHandler.java:204) at com.sun.jersey.client.urlconnection.URLConnectionClientHandler.handle(URLConnectionClientHandler.java:147) ... 9 moreit>(HttpClient.java:211) at sun.net.www.http.HttpClient.New(HttpClient.java:308) at sun.net.www.http.HttpClient.New(HttpClient.java:326) at sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:996) at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:932) at sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:850) at sun.net.www.protocol.http.HttpURLConnection.getOutputStream(HttpURLConnection.java:1091) at com.sun.jersey.client.urlconnection.URLConnectionClientHandler$1$1.getOutputStream(URLConnectionClientHandler.java:225) at com.sun.jersey.api.client.CommittingOutputStream.commitWrite(CommittingOutputStream.java:117) at com.sun.jersey.api.client.CommittingOutputStream.write(CommittingOutputStream.java:89) at org.codehaus.jackson.impl.Utf8Generator._flushBuffer(Utf8Generator.java:1754) at org.codehaus.jackson.impl.Utf8Generator.flush(Utf8Generator.java:1088) at org.codehaus.jackson.map.ObjectMapper.writeValue(ObjectMapper.java:1354) at org.codehaus.jackson.jaxrs.JacksonJsonProvider.writeTo(JacksonJsonProvider.java:527) at com.sun.jersey.api.client.RequestWriter.writeRequestEntity(RequestWriter.java:300) at com.sun.jersey.client.urlconnection.URLConnectionClientHandler._invoke(URLConnectionClientHandler.java:204) at com.sun.jersey.client.urlconnection.URLConnectionClientHandler.handle(URLConnectionClientHandler.java:147) ... 9 more {noformat}

• YARN-1957. Major sub-task reported by Carlo Curino and fixed by Carlo Curino (resourcemanager)
  ProportionalCapacitPreemptionPolicy handling of corner cases...
  

  The current version of ProportionalCapacityPreemptionPolicy should be improved to deal with the following two scenarios: 1) when rebalancing over-capacity allocations, it potentially preempts without considering the maxCapacity constraints of a queue (i.e., preempting possibly more than strictly necessary) 2) a zero capacity queue is preempted even if there is no demand (coherent with old use of zero-capacity to disabled queues) The proposed patch fixes both issues, and introduce few new test cases.

• YARN-1947. Major test reported by Jian He and fixed by Jian He
  TestRMDelegationTokens#testRMDTMasterKeyStateOnRollingMasterKey is failing intermittently
  

  java.lang.AssertionError: null at org.junit.Assert.fail(Assert.java:92) at org.junit.Assert.assertTrue(Assert.java:43) at org.junit.Assert.assertTrue(Assert.java:54) at org.apache.hadoop.yarn.server.resourcemanager.security.TestRMDelegationTokens.testRMDTMasterKeyStateOnRollingMasterKey(TestRMDelegationTokens.java:117)

• YARN-1934. Blocker bug reported by Rohith and fixed by Karthik Kambatla (resourcemanager)
  Potential NPE in ZKRMStateStore caused by handling Disconnected event from ZK.
  

  For ZK disconnected event , zkClient is set to null. It is very much prone to throw NPE. {noformat} case Disconnected: LOG.info("ZKRMStateStore Session disconnected"); oldZkClient = zkClient; zkClient = null; break; {noformat}

• YARN-1933. Major bug reported by Jian He and fixed by Jian He
  TestAMRestart and TestNodeHealthService failing sometimes on Windows
  

  TestNodeHealthService failures: testNodeHealthScript(org.apache.hadoop.yarn.server.nodemanager.TestNodeHealthService) Time elapsed: 1.405 sec <<< ERROR! java.io.FileNotFoundException: C:\Users\Administrator\Documents\hadoop-common\hadoop-yarn-project\hadoop-yarn\hadoop-yarn-server\hadoop-yarn-server-nodemanager\target\org.apache.hadoop.yarn.server.nodemanager.TestNodeHealthService-localDir\failingscript.cmd (The process cannot access the file because it is being used by another process) at java.io.FileOutputStream.open(Native Method) at java.io.FileOutputStream.<init>(FileOutputStream.java:221) at java.io.FileOutputStream.<init>(FileOutputStream.java:171) at org.apache.hadoop.yarn.server.nodemanager.TestNodeHealthService.writeNodeHealthScriptFile(TestNodeHealthService.java:82) at org.apache.hadoop.yarn.server.nodemanager.TestNodeHealthService.testNodeHealthScript(TestNodeHealthService.java:154) testNodeHealthScriptShouldRun(org.apache.hadoop.yarn.server.nodemanager.TestNodeHealthService) Time elapsed: 0 sec <<< ERROR! java.io.FileNotFoundException: C:\Users\Administrator\Documents\hadoop-common\hadoop-yarn-project\hadoop-yarn\hadoop-yarn-server\hadoop-yarn-server-nodemanager\target\org.apache.hadoop.yarn.server.nodemanager.TestNodeHealthService-localDir\failingscript.cmd (Access is denied) at java.io.FileOutputStream.open(Native Method) at java.io.FileOutputStream.<init>(FileOutputStream.java:221) at java.io.FileOutputStream.<init>(FileOutputStream.java:171) at org.apache.hadoop.yarn.server.nodemanager.TestNodeHealthService.writeNodeHealthScriptFile(TestNodeHealthService.java:82) at org.apache.hadoop.yarn.server.nodemanager.TestNodeHealthService.testNodeHealthScriptShouldRun(TestNodeHealthService.java:103)

• YARN-1932. Blocker bug reported by Mit Desai and fixed by Mit Desai
  Javascript injection on the job status page
  

  Scripts can be injected into the job status page as the diagnostics field is not sanitized. Whatever string you set there will show up to the jobs page as it is ... ie. if you put any script commands, they will be executed in the browser of the user who is opening the page. We need escaping the diagnostic string in order to not run the scripts.

• YARN-1931. Blocker bug reported by Thomas Graves and fixed by Sandy Ryza (applications)
  Private API change in YARN-1824 in 2.4 broke compatibility with previous releases
  

  YARN-1824 broke compatibility with previous 2.x releases by changes the API's in org.apache.hadoop.yarn.util.Apps.{setEnvFromInputString,addToEnvironment} The old api should be added back in. This affects any ApplicationMasters who were using this api. It also breaks previously built MapReduce libraries from working with the new Yarn release as MR uses this api.

• YARN-1929. Blocker bug reported by Rohith and fixed by Karthik Kambatla (resourcemanager)
  DeadLock in RM when automatic failover is enabled.
  

  Dead lock detected in RM when automatic failover is enabled. {noformat} Found one Java-level deadlock: ============================= "Thread-2": waiting to lock monitor 0x00007fb514303cf0 (object 0x00000000ef153fd0, a org.apache.hadoop.ha.ActiveStandbyElector), which is held by "main-EventThread" "main-EventThread": waiting to lock monitor 0x00007fb514750a48 (object 0x00000000ef154020, a org.apache.hadoop.yarn.server.resourcemanager.EmbeddedElectorService), which is held by "Thread-2" {noformat}

• YARN-1928. Major bug reported by Zhijie Shen and fixed by Zhijie Shen
  TestAMRMRPCNodeUpdates fails ocassionally
  

  {code} junit.framework.AssertionFailedError: expected:<0> but was:<4> at junit.framework.Assert.fail(Assert.java:50) at junit.framework.Assert.failNotEquals(Assert.java:287) at junit.framework.Assert.assertEquals(Assert.java:67) at junit.framework.Assert.assertEquals(Assert.java:199) at junit.framework.Assert.assertEquals(Assert.java:205) at org.apache.hadoop.yarn.server.resourcemanager.applicationsmanager.TestAMRMRPCNodeUpdates.testAMRMUnusableNodes(TestAMRMRPCNodeUpdates.java:136) {code}

• YARN-1926. Major bug reported by Varun Vasudev and fixed by Varun Vasudev
  DistributedShell unit tests fail on Windows
  

  Couple of unit tests for the DistributedShell fail on Windows - specifically testDSShellWithShellScript and testDSRestartWithPreviousRunningContainers

• YARN-1924. Critical bug reported by Arpit Gupta and fixed by Jian He
  STATE_STORE_OP_FAILED happens when ZKRMStateStore tries to update app(attempt) before storing it
  

  Noticed on a HA cluster Both RM shut down with this error.

• YARN-1920. Major bug reported by Vinod Kumar Vavilapalli and fixed by Vinod Kumar Vavilapalli
  TestFileSystemApplicationHistoryStore.testMissingApplicationAttemptHistoryData fails in windows
  

  Though this was only failing in Windows, after debugging, I realized that the test fails because we are leaking a file-handle in the history service.

• YARN-1914. Major bug reported by Varun Vasudev and fixed by Varun Vasudev
  Test TestFSDownload.testDownloadPublicWithStatCache fails on Windows
  

  The TestFSDownload.testDownloadPublicWithStatCache test in hadoop-yarn-common consistently fails on Windows environments. The root cause is that the test checks for execute permission for all users on every ancestor of the target directory. In windows, by default, group "Everyone" has no permissions on any directory in the install drive. It's unreasonable to expect this test to pass and we should skip it on Windows.

• YARN-1910. Major bug reported by Xuan Gong and fixed by Xuan Gong
  TestAMRMTokens fails on windows
  

• YARN-1908. Major bug reported by Tassapol Athiapinya and fixed by Vinod Kumar Vavilapalli (applications/distributed-shell)
  Distributed shell with custom script has permission error.
  

  Create test1.sh having "pwd". Run this command as user1: hadoop jar /usr/lib/hadoop-yarn/hadoop-yarn-applications-distributedshell.jar -jar /usr/lib/hadoop-yarn/hadoop-yarn-applications-distributedshell.jar -shell_script test1.sh NM is run by yarn user. An exception is thrown because yarn user has no permissions on custom script in hdfs path. The custom script is created with distributed shell app. {code} Caused by: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.AccessControlException): Permission denied: user=yarn, access=WRITE, inode="/user/user1/DistributedShell/70":user1:user1:drwxr-xr-x at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkFsPermission(FSPermissionChecker.java:265) {code}

• YARN-1907. Major bug reported by Mit Desai and fixed by Mit Desai
  TestRMApplicationHistoryWriter#testRMWritingMassiveHistory runs slow and intermittently fails
  

  The test has 10000 containers that it tries to cleanup. The cleanup has a timeout of 20000ms in which the test sometimes cannot do the cleanup completely and gives out an Assertion Failure.

• YARN-1905. Trivial test reported by Chris Nauroth and fixed by Chris Nauroth (nodemanager)
  TestProcfsBasedProcessTree must only run on Linux.
  

  The tests in {{TestProcfsBasedProcessTree}} only make sense on Linux, where the process tree calculations are based on reading the /proc file system. Right now, not all of the individual tests are skipped when the OS is not Linux. This patch will make it consistent.

• YARN-1903. Major bug reported by Zhijie Shen and fixed by Zhijie Shen
  Killing Container on NEW and LOCALIZING will result in exitCode and diagnostics not set
  

  The container status after stopping container is not expected. {code} java.lang.AssertionError: 4: at org.junit.Assert.fail(Assert.java:93) at org.junit.Assert.assertTrue(Assert.java:43) at org.apache.hadoop.yarn.client.api.impl.TestNMClient.testGetContainerStatus(TestNMClient.java:382) at org.apache.hadoop.yarn.client.api.impl.TestNMClient.testContainerManagement(TestNMClient.java:346) at org.apache.hadoop.yarn.client.api.impl.TestNMClient.testNMClient(TestNMClient.java:226) {code}

• YARN-1898. Major sub-task reported by Yesha Vora and fixed by Xuan Gong (resourcemanager)
  Standby RM's conf, stacks, logLevel, metrics, jmx and logs links are redirecting to Active RM
  

  Standby RM links /conf, /stacks, /logLevel, /metrics, /jmx is redirected to Active RM. It should not be redirected to Active RM

• YARN-1892. Minor improvement reported by Siddharth Seth and fixed by Jian He (scheduler)
  Excessive logging in RM
  

  Mostly in the CS I believe {code} INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerApplicationAttempt: Application application_1395435468498_0011 reserved container container_1395435468498_0011_01_000213 on node host: #containers=5 available=4096 used=20960, currently has 1 at priority 4; currentReservation 4096 {code} {code} INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue: hive2 usedResources: <memory:20480, vCores:5> clusterResources: <memory:81920, vCores:16> currentCapacity 0.25 required <memory:4096, vCores:1> potentialNewCapacity: 0.255 ( max-capacity: 0.25) {code}

• YARN-1883. Major bug reported by Mit Desai and fixed by Mit Desai
  TestRMAdminService fails due to inconsistent entries in UserGroups
  

  testRefreshUserToGroupsMappingsWithFileSystemBasedConfigurationProvider fails with the following error: {noformat} java.lang.AssertionError: null at org.junit.Assert.fail(Assert.java:92) at org.junit.Assert.assertTrue(Assert.java:43) at org.junit.Assert.assertTrue(Assert.java:54) at org.apache.hadoop.yarn.server.resourcemanager.TestRMAdminService.testRefreshUserToGroupsMappingsWithFileSystemBasedConfigurationProvider(TestRMAdminService.java:421) at org.apache.hadoop.yarn.server.resourcemanager.TestRMAdminService.testOrder(TestRMAdminService.java:104) {noformat} Line Numbers will be inconsistent as I was testing to run it in a particular order. But the Line on which the failure occurs is {code} Assert.assertTrue(groupBefore.contains("test_group_A") && groupBefore.contains("test_group_B") && groupBefore.contains("test_group_C") && groupBefore.size() == 3); {code} testRMInitialsWithFileSystemBasedConfigurationProvider() and testRefreshUserToGroupsMappingsWithFileSystemBasedConfigurationProvider() calls the function {{MockUnixGroupsMapping.updateGroups();}} which changes the list of userGroups. testRefreshUserToGroupsMappingsWithFileSystemBasedConfigurationProvider() tries to verify the groups before changing it and fails if testRMInitialsWithFileSystemBasedConfigurationProvider() already ran and made the changes.

• YARN-1861. Blocker sub-task reported by Arpit Gupta and fixed by Karthik Kambatla (resourcemanager)
  Both RM stuck in standby mode when automatic failover is enabled
  

  In our HA tests we noticed that the tests got stuck because both RM's got into standby state and no one became active.

• YARN-1837. Major bug reported by Tsuyoshi OZAWA and fixed by Hong Zhiguo
  TestMoveApplication.testMoveRejectedByScheduler randomly fails
  

  TestMoveApplication#testMoveRejectedByScheduler fails because of NullPointerException. It looks caused by unhandled exception handling at server-side.

• YARN-1750. Major test reported by Ming Ma and fixed by Wangda Tan (nodemanager)
  TestNodeStatusUpdater#testNMRegistration is incorrect in test case
  

  This test case passes. However, the test output log has java.lang.AssertionError: Number of applications should only be one! expected:<1> but was:<2> at org.junit.Assert.fail(Assert.java:93) at org.junit.Assert.failNotEquals(Assert.java:647) at org.junit.Assert.assertEquals(Assert.java:128) at org.junit.Assert.assertEquals(Assert.java:472) at org.apache.hadoop.yarn.server.nodemanager.TestNodeStatusUpdater$MyResourceTracker.nodeHeartbeat(TestNodeStatusUpdater.java:267) at org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl$1.run(NodeStatusUpdaterImpl.java:469) at java.lang.Thread.run(Thread.java:695) TestNodeStatusUpdater.java has invalid asserts. } else if (heartBeatID == 3) { // Checks on the RM end Assert.assertEquals("Number of applications should only be one!", 1, appToContainers.size()); Assert.assertEquals("Number of container for the app should be two!", 2, appToContainers.get(appId2).size()); We should fix the assert and add more check to the test.

• YARN-1701. Major sub-task reported by Gera Shegalov and fixed by Tsuyoshi OZAWA
  Improve default paths of timeline store and generic history store
  

  When I enable AHS via yarn.ahs.enabled, the app history is still not visible in AHS webUI. This is due to NullApplicationHistoryStore as yarn.resourcemanager.history-writer.class. It would be good to have just one key to enable basic functionality. yarn.ahs.fs-history-store.uri uses {code}${hadoop.log.dir}{code}, which is local file system location. However, FileSystemApplicationHistoryStore uses DFS by default.

• YARN-1696. Blocker sub-task reported by Karthik Kambatla and fixed by Tsuyoshi OZAWA (resourcemanager)
  Document RM HA
  

  Add documentation for RM HA. Marking this a blocker for 2.4 as this is required to call RM HA Stable and ready for public consumption.

• YARN-1281. Major test reported by Karthik Kambatla and fixed by Tsuyoshi OZAWA (resourcemanager)
  TestZKRMStateStoreZKClientConnections fails intermittently
  

  The test fails intermittently - haven't been able to reproduce the failure deterministically.

• YARN-1201. Minor bug reported by Nemon Lou and fixed by Wangda Tan (resourcemanager)
  TestAMAuthorization fails with local hostname cannot be resolved
  

  When hostname is 158-1-131-10, TestAMAuthorization fails. {code} Running org.apache.hadoop.yarn.server.resourcemanager.TestAMAuthorization Tests run: 4, Failures: 0, Errors: 2, Skipped: 0, Time elapsed: 14.034 sec <<< FAILURE! - in org.apache.hadoop.yarn.server.resourcemanager.TestAMAuthorization testUnauthorizedAccess[0](org.apache.hadoop.yarn.server.resourcemanager.TestAMAuthorization) Time elapsed: 3.952 sec <<< ERROR! java.lang.NullPointerException: null at org.apache.hadoop.yarn.server.resourcemanager.TestAMAuthorization.testUnauthorizedAccess(TestAMAuthorization.java:284) testUnauthorizedAccess[1](org.apache.hadoop.yarn.server.resourcemanager.TestAMAuthorization) Time elapsed: 3.116 sec <<< ERROR! java.lang.NullPointerException: null at org.apache.hadoop.yarn.server.resourcemanager.TestAMAuthorization.testUnauthorizedAccess(TestAMAuthorization.java:284) Results : Tests in error: TestAMAuthorization.testUnauthorizedAccess:284 NullPointer TestAMAuthorization.testUnauthorizedAccess:284 NullPointer Tests run: 4, Failures: 0, Errors: 2, Skipped: 0 {code}

• MAPREDUCE-5843. Major test reported by Varun Vasudev and fixed by Varun Vasudev
  TestMRKeyValueTextInputFormat failing on Windows
  

• MAPREDUCE-5841. Major bug reported by Sangjin Lee and fixed by Sangjin Lee (mrv2)
  uber job doesn't terminate on getting mapred job kill
  

• MAPREDUCE-5835. Critical bug reported by Ming Ma and fixed by Ming Ma
  Killing Task might cause the job to go to ERROR state
  

• MAPREDUCE-5833. Major test reported by Zhijie Shen and fixed by Zhijie Shen
  TestRMContainerAllocator fails ocassionally
  

• MAPREDUCE-5832. Major bug reported by Jian He and fixed by Vinod Kumar Vavilapalli
  Few tests in TestJobClient fail on Windows
  

• MAPREDUCE-5830. Blocker bug reported by Jason Lowe and fixed by Akira AJISAKA
  HostUtil.getTaskLogUrl is not backwards binary compatible with 2.3
  

• MAPREDUCE-5828. Major bug reported by Vinod Kumar Vavilapalli and fixed by Vinod Kumar Vavilapalli
  TestMapReduceJobControl fails on JDK 7 + Windows
  

• MAPREDUCE-5827. Major bug reported by Zhijie Shen and fixed by Zhijie Shen
  TestSpeculativeExecutionWithMRApp fails
  

• MAPREDUCE-5826. Major bug reported by Varun Vasudev and fixed by Varun Vasudev
  TestHistoryServerFileSystemStateStoreService.testTokenStore fails in windows
  

• MAPREDUCE-5824. Major bug reported by Xuan Gong and fixed by Xuan Gong
  TestPipesNonJavaInputFormat.testFormat fails in windows
  

• MAPREDUCE-5821. Major bug reported by Todd Lipcon and fixed by Todd Lipcon (performance , task)
  IFile merge allocates new byte array for every value
  

• MAPREDUCE-5818. Major bug reported by Jian He and fixed by Jian He
  hsadmin cmd is missing in mapred.cmd
  

• MAPREDUCE-5815. Blocker bug reported by Gera Shegalov and fixed by Akira AJISAKA (client , mrv2)
  Fix NPE in TestMRAppMaster
  

• MAPREDUCE-5714. Major bug reported by Jinghui Wang and fixed by Jinghui Wang (test)
  TestMRAppComponentDependencies causes surefire to exit without saying proper goodbye
  

• MAPREDUCE-3191. Trivial bug reported by Todd Lipcon and fixed by Chen He
  docs for map output compression incorrectly reference SequenceFile
  

• HDFS-6527. Blocker bug reported by Kihwal Lee and fixed by Kihwal Lee
  Edit log corruption due to defered INode removal
  

• HDFS-6411. Major bug reported by Zhongyi Xie and fixed by Brandon Li (nfs)
  nfs-hdfs-gateway mount raises I/O error and hangs when a unauthorized user attempts to access it
  

• HDFS-6402. Trivial bug reported by Chris Nauroth and fixed by Chris Nauroth (namenode)
  Suppress findbugs warning for failure to override equals and hashCode in FsAclPermission.
  

• HDFS-6397. Critical bug reported by Mohammad Kamrul Islam and fixed by Mohammad Kamrul Islam
  NN shows inconsistent value in deadnode count
  

• HDFS-6362. Blocker bug reported by Arpit Agarwal and fixed by Arpit Agarwal (namenode)
  InvalidateBlocks is inconsistent in usage of DatanodeUuid and StorageID
  

• HDFS-6361. Major bug reported by Yongjun Zhang and fixed by Yongjun Zhang (nfs)
  TestIdUserGroup.testUserUpdateSetting failed due to out of range nfsnobody Id
  

• HDFS-6340. Blocker bug reported by Rahul Singhal and fixed by Rahul Singhal (datanode)
  DN can't finalize upgrade
  

• HDFS-6329. Blocker bug reported by Kihwal Lee and fixed by Kihwal Lee
  WebHdfs does not work if HA is enabled on NN but logical URI is not configured.
  

• HDFS-6326. Blocker bug reported by Daryn Sharp and fixed by Chris Nauroth (webhdfs)
  WebHdfs ACL compatibility is broken
  

• HDFS-6325. Major bug reported by Konstantin Shvachko and fixed by Keith Pak (namenode)
  Append should fail if the last block has insufficient number of replicas
  

  I have committed the fix to the trunk, branch-2, and branch-2.4 respectively. Thanks Keith!

• HDFS-6313. Blocker bug reported by Daryn Sharp and fixed by Kihwal Lee (webhdfs)
  WebHdfs may use the wrong NN when configured for multiple HA NNs
  

• HDFS-6245. Major bug reported by Arpit Gupta and fixed by Arpit Agarwal
  datanode fails to start with a bad disk even when failed volumes is set
  

• HDFS-6236. Minor bug reported by Chris Nauroth and fixed by Chris Nauroth (namenode)
  ImageServlet should use Time#monotonicNow to measure latency.
  

• HDFS-6235. Trivial bug reported by Chris Nauroth and fixed by Chris Nauroth (namenode , test)
  TestFileJournalManager can fail on Windows due to file locking if tests run out of order.
  

• HDFS-6234. Trivial bug reported by Chris Nauroth and fixed by Chris Nauroth (datanode , test)
  TestDatanodeConfig#testMemlockLimit fails on Windows due to invalid file path.
  

• HDFS-6232. Major bug reported by Stephen Chu and fixed by Akira AJISAKA (tools)
  OfflineEditsViewer throws a NPE on edits containing ACL modifications
  

• HDFS-6231. Major bug reported by Chris Nauroth and fixed by Chris Nauroth (hdfs-client)
  DFSClient hangs infinitely if using hedged reads and all eligible datanodes die.
  

• HDFS-6229. Major bug reported by Jing Zhao and fixed by Jing Zhao (ha)
  Race condition in failover can cause RetryCache fail to work
  

• HDFS-6215. Minor bug reported by Kihwal Lee and fixed by Kihwal Lee
  Wrong error message for upgrade
  

• HDFS-6209. Minor bug reported by Arpit Agarwal and fixed by Arpit Agarwal (test)
  Fix flaky test TestValidateConfigurationSettings.testThatDifferentRPCandHttpPortsAreOK
  

• HDFS-6208. Major bug reported by Chris Nauroth and fixed by Chris Nauroth (datanode)
  DataNode caching can leak file descriptors.
  

• HDFS-6206. Major bug reported by Tsz Wo Nicholas Sze and fixed by Tsz Wo Nicholas Sze
  DFSUtil.substituteForWildcardAddress may throw NPE
  

• HDFS-6204. Minor bug reported by Tsz Wo Nicholas Sze and fixed by Tsz Wo Nicholas Sze (test)
  TestRBWBlockInvalidation may fail
  

• HDFS-6198. Major bug reported by Chris Nauroth and fixed by Chris Nauroth (datanode)
  DataNode rolling upgrade does not correctly identify current block pool directory and replace with trash on Windows.
  

• HDFS-6197. Minor bug reported by Chris Nauroth and fixed by Chris Nauroth (namenode)
  Rolling upgrade rollback on Windows can fail attempting to rename edit log segment files to a destination that already exists.
  

• HDFS-6189. Major test reported by Chris Nauroth and fixed by Chris Nauroth (test)
  Multiple HDFS tests fail on Windows attempting to use a test root path containing a colon.
  

• HDFS-4052. Minor improvement reported by Jing Zhao and fixed by Jing Zhao
  BlockManager#invalidateWork should print logs outside the lock
  

• HDFS-2882. Major bug reported by Todd Lipcon and fixed by Vinayakumar B (datanode)
  DN continues to start up, even if block pool fails to initialize
  

• HADOOP-10612. Major bug reported by Brandon Li and fixed by Brandon Li (nfs)
  NFS failed to refresh the user group id mapping table
  

• HADOOP-10562. Critical bug reported by Suresh Srinivas and fixed by Suresh Srinivas
  Namenode exits on exception without printing stack trace in AbstractDelegationTokenSecretManager
  

• HADOOP-10527. Major bug reported by Kihwal Lee and fixed by Kihwal Lee
  Fix incorrect return code and allow more retries on EINTR
  

• HADOOP-10522. Critical bug reported by Kihwal Lee and fixed by Kihwal Lee
  JniBasedUnixGroupMapping mishandles errors
  

• HADOOP-10490. Minor bug reported by Chris Nauroth and fixed by Chris Nauroth (test)
  TestMapFile and TestBloomMapFile leak file descriptors.
  

• HADOOP-10473. Minor bug reported by Tsz Wo Nicholas Sze and fixed by Tsz Wo Nicholas Sze (test)
  TestCallQueueManager is still flaky
  

• HADOOP-10466. Minor improvement reported by Nicolas Liochon and fixed by Nicolas Liochon (security)
  Lower the log level in UserGroupInformation
  

• HADOOP-10456. Major bug reported by Nishkam Ravi and fixed by Nishkam Ravi (conf)
  Bug in Configuration.java exposed by Spark (ConcurrentModificationException)
  

• HADOOP-10455. Major bug reported by Tsz Wo Nicholas Sze and fixed by Tsz Wo Nicholas Sze (ipc)
  When there is an exception, ipc.Server should first check whether it is an terse exception
  

• HADOOP-8826. Minor bug reported by Robert Joseph Evans and fixed by Mit Desai
  Docs still refer to 0.20.205 as stable line
  

Hadoop 2.4.0 Release Notes

• YARN-1893. Blocker sub-task reported by Xuan Gong and fixed by Xuan Gong (resourcemanager)
  Make ApplicationMasterProtocol#allocate AtMostOnce
  

• YARN-1891. Minor task reported by Varun Vasudev and fixed by Varun Vasudev
  Document NodeManager health-monitoring
  

  Start documenting node manager starting with the health monitoring.

• YARN-1873. Major bug reported by Mit Desai and fixed by Mit Desai
  TestDistributedShell#testDSShell fails when the test cases are out of order
  

  testDSShell fails when the tests are run in random order. I see a cleanup issue here. {noformat} Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 72.222 sec <<< FAILURE! - in org.apache.hadoop.yarn.applications.distributedshell.TestDistributedShell testOrder(org.apache.hadoop.yarn.applications.distributedshell.TestDistributedShell) Time elapsed: 44.127 sec <<< FAILURE! java.lang.AssertionError: expected:<1> but was:<6> at org.junit.Assert.fail(Assert.java:93) at org.junit.Assert.failNotEquals(Assert.java:647) at org.junit.Assert.assertEquals(Assert.java:128) at org.junit.Assert.assertEquals(Assert.java:472) at org.junit.Assert.assertEquals(Assert.java:456) at org.apache.hadoop.yarn.applications.distributedshell.TestDistributedShell.testDSShell(TestDistributedShell.java:204) at org.apache.hadoop.yarn.applications.distributedshell.TestDistributedShell.testOrder(TestDistributedShell.java:134) Results : Failed tests: TestDistributedShell.testOrder:134->testDSShell:204 expected:<1> but was:<6> {noformat} The Line numbers will be little deviated because I was trying to reproduce the error by running the tests in specific order. But the Line that causes the assert fail is {{Assert.assertEquals(1, entitiesAttempts.getEntities().size());}}

• YARN-1867. Blocker bug reported by Karthik Kambatla and fixed by Vinod Kumar Vavilapalli (resourcemanager)
  NPE while fetching apps via the REST API
  

  We ran into the following NPE when fetching applications using the REST API: {noformat} INTERNAL_SERVER_ERROR java.lang.NullPointerException at org.apache.hadoop.yarn.server.security.ApplicationACLsManager.checkAccess(ApplicationACLsManager.java:104) at org.apache.hadoop.yarn.server.resourcemanager.webapp.RMWebServices.hasAccess(RMWebServices.java:123) at org.apache.hadoop.yarn.server.resourcemanager.webapp.RMWebServices.getApps(RMWebServices.java:418) {noformat}

• YARN-1866. Blocker bug reported by Arpit Gupta and fixed by Jian He
  YARN RM fails to load state store with delegation token parsing error
  

  In our secure Nightlies we saw exceptions in the RM log where it failed to parse the deletegation token.

• YARN-1863. Blocker test reported by Ted Yu and fixed by Xuan Gong
  TestRMFailover fails with 'AssertionError: null'
  

  This happened in Hadoop-Yarn-trunk - Build # 516 and can be reproduced: {code} testWebAppProxyInStandAloneMode(org.apache.hadoop.yarn.client.TestRMFailover) Time elapsed: 5.834 sec <<< FAILURE! java.lang.AssertionError: null at org.junit.Assert.fail(Assert.java:92) at org.junit.Assert.assertTrue(Assert.java:43) at org.junit.Assert.assertTrue(Assert.java:54) at org.apache.hadoop.yarn.client.TestRMFailover.verifyExpectedException(TestRMFailover.java:250) at org.apache.hadoop.yarn.client.TestRMFailover.testWebAppProxyInStandAloneMode(TestRMFailover.java:216) testEmbeddedWebAppProxy(org.apache.hadoop.yarn.client.TestRMFailover) Time elapsed: 5.341 sec <<< FAILURE! java.lang.AssertionError: null at org.junit.Assert.fail(Assert.java:92) at org.junit.Assert.assertTrue(Assert.java:43) at org.junit.Assert.assertTrue(Assert.java:54) at org.apache.hadoop.yarn.client.TestRMFailover.verifyExpectedException(TestRMFailover.java:250) at org.apache.hadoop.yarn.client.TestRMFailover.testEmbeddedWebAppProxy(TestRMFailover.java:241) {code}

• YARN-1859. Major bug reported by Zhijie Shen and fixed by Zhijie Shen
  WebAppProxyServlet will throw ApplicationNotFoundException if the app is no longer cached in RM
  

  WebAppProxyServlet checks null to determine whether the application is not found or not. {code} ApplicationReport applicationReport = getApplicationReport(id); if(applicationReport == null) { LOG.warn(req.getRemoteUser()+" Attempting to access "+id+ " that was not found"); {code} However, WebAppProxyServlet calls AppReportFetcher, which consequently calls ClientRMService. When application is not found, ClientRMService throws ApplicationNotFoundException. Therefore, in WebAppProxyServlet, the following logic to create the tracking url for a non-cached app will no longer be in use.

• YARN-1855. Critical test reported by Ted Yu and fixed by Zhijie Shen
  TestRMFailover#testRMWebAppRedirect fails in trunk
  

  From https://builds.apache.org/job/Hadoop-Yarn-trunk/514/console : {code} testRMWebAppRedirect(org.apache.hadoop.yarn.client.TestRMFailover) Time elapsed: 5.39 sec <<< ERROR! java.lang.NullPointerException: null at org.apache.hadoop.yarn.client.TestRMFailover.testRMWebAppRedirect(TestRMFailover.java:269) {code}

• YARN-1854. Blocker test reported by Mit Desai and fixed by Rohith
  Race condition in TestRMHA#testStartAndTransitions
  

  There is race in test. TestRMHA#testStartAndTransitions calls verifyClusterMetrics() immediately after application is submitted, but QueueMetrics are updated after app attempt is sheduled. Calling verifyClusterMetrics() without verifying app attempt is in Scheduled state cause random test failures. MockRM.submitApp() return when application is in ACCEPTED, but QueueMetrics updated at APP_ATTEMPT_ADDED event. There is high chance of getting queue metrics before app attempt is Scheduled. {noformat} testStartAndTransitions(org.apache.hadoop.yarn.server.resourcemanager.TestRMHA) Time elapsed: 5.883 sec <<< FAILURE! java.lang.AssertionError: Incorrect value for metric availableMB expected:<2048> but was:<4096> at org.junit.Assert.fail(Assert.java:93) at org.junit.Assert.failNotEquals(Assert.java:647) at org.junit.Assert.assertEquals(Assert.java:128) at org.junit.Assert.assertEquals(Assert.java:472) at org.apache.hadoop.yarn.server.resourcemanager.TestRMHA.assertMetric(TestRMHA.java:396) at org.apache.hadoop.yarn.server.resourcemanager.TestRMHA.verifyClusterMetrics(TestRMHA.java:387) at org.apache.hadoop.yarn.server.resourcemanager.TestRMHA.testStartAndTransitions(TestRMHA.java:160) Results : Failed tests: TestRMHA.testStartAndTransitions:160->verifyClusterMetrics:387->assertMetric:396 Incorrect value for metric availableMB expected:<2048> but was:<4096> {noformat}

• YARN-1852. Major bug reported by Rohith and fixed by Rohith (resourcemanager)
  Application recovery throws InvalidStateTransitonException for FAILED and KILLED jobs
  

  Recovering for failed/killed application throw InvalidStateTransitonException. These are logged during recovery of applications.

• YARN-1850. Major sub-task reported by Zhijie Shen and fixed by Zhijie Shen
  Make enabling timeline service configurable
  

  Like generic history service, we'd better to make enabling timeline service configurable, in case the timeline server is not up

• YARN-1849. Blocker bug reported by Karthik Kambatla and fixed by Karthik Kambatla (resourcemanager)
  NPE in ResourceTrackerService#registerNodeManager for UAM
  

  While running an UnmanagedAM on secure cluster, ran into an NPE on failover/restart. This is similar to YARN-1821.

• YARN-1846. Major bug reported by Robert Kanter and fixed by Robert Kanter
  TestRM#testNMTokenSentForNormalContainer assumes CapacityScheduler
  

  TestRM.testNMTokenSentForNormalContainer assumes the CapacityScheduler is being used and tries to do: {code:java} CapacityScheduler cs = (CapacityScheduler) rm.getResourceScheduler(); {code} This throws a {{ClassCastException}} if you're not using the CapacityScheduler.

• YARN-1839. Critical bug reported by Tassapol Athiapinya and fixed by Jian He (applications , capacityscheduler)
  Capacity scheduler preempts an AM out. AM attempt 2 fails to launch task container with SecretManager$InvalidToken: No NMToken sent
  

  Use single-node cluster. Turn on capacity scheduler preemption. Run MR sleep job as app 1. Take entire cluster. Run MR sleep job as app 2. Preempt app1 out. Wait till app 2 finishes. App 1 AM attempt 2 will start. It won't be able to launch a task container with this error stack trace in AM logs: {code} 2014-03-13 20:13:50,254 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Diagnostics report from attempt_1394741557066_0001_m_000000_1009: Container launch failed for container_1394741557066_0001_02_000021 : org.apache.hadoop.security.token.SecretManager$InvalidToken: No NMToken sent for <host>:45454 at org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy$ContainerManagementProtocolProxyData.newProxy(ContainerManagementProtocolProxy.java:206) at org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy$ContainerManagementProtocolProxyData.<init>(ContainerManagementProtocolProxy.java:196) at org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy.getProxy(ContainerManagementProtocolProxy.java:117) at org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl.getCMProxy(ContainerLauncherImpl.java:403) at org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$Container.launch(ContainerLauncherImpl.java:138) at org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$EventProcessor.run(ContainerLauncherImpl.java:369) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:722) {code}

• YARN-1838. Major sub-task reported by Srimanth Gunturi and fixed by Billie Rinaldi
  Timeline service getEntities API should provide ability to get entities from given id
  

  To support pagination, we need ability to get entities from a certain ID by providing a new param called {{fromid}}. For example on a page of 10 jobs, our first call will be like [http://server:8188/ws/v1/timeline/HIVE_QUERY_ID?fields=events,primaryfilters,otherinfo&limit=11] When user hits next, we would like to call [http://server:8188/ws/v1/timeline/HIVE_QUERY_ID?fields=events,primaryfilters,otherinfo&fromid=JID11&limit=11] and continue on for further _Next_ clicks On hitting back, we will make similar calls for previous items [http://server:8188/ws/v1/timeline/HIVE_QUERY_ID?fields=events,primaryfilters,otherinfo&fromid=JID1&limit=11] {{fromid}} should be inclusive of the id given.

• YARN-1833. Major bug reported by Mit Desai and fixed by Mit Desai
  TestRMAdminService Fails in trunk and branch-2 : Assert Fails due to different count of UserGroups for currentUser()
  

  In the test testRefreshUserToGroupsMappingsWithFileSystemBasedConfigurationProvider, the following assert is not needed. {code} Assert.assertTrue(groupWithInit.size() != groupBefore.size()); {code} As the assert takes the default groups for groupWithInit (which in my case are users, sshusers and wheel), it fails as the size of both groupWithInit and groupBefore are same. I do not think we need to have this assert here. Moreover we are also checking that the groupInit does not have the userGroups that are in the groupBefore so removing the assert may not be harmful.

• YARN-1830. Major bug reported by Karthik Kambatla and fixed by Zhijie Shen (resourcemanager)
  TestRMRestart.testQueueMetricsOnRMRestart failure
  

  TestRMRestart.testQueueMetricsOnRMRestart fails intermittently as follows (reported on YARN-1815): {noformat} java.lang.AssertionError: expected:<37> but was:<38> ... at org.apache.hadoop.yarn.server.resourcemanager.TestRMRestart.assertQueueMetrics(TestRMRestart.java:1728) at org.apache.hadoop.yarn.server.resourcemanager.TestRMRestart.testQueueMetricsOnRMRestart(TestRMRestart.java:1682) {noformat}

• YARN-1824. Major bug reported by Jian He and fixed by Jian He
  Make Windows client work with Linux/Unix cluster
  

• YARN-1821. Blocker sub-task reported by Karthik Kambatla and fixed by Karthik Kambatla (resourcemanager)
  NPE on registerNodeManager if the request has containers for UnmanagedAMs
  

  On RM restart (or failover), NM re-registers with the RM. If it was running containers for Unmanaged AMs, it runs into the following NPE: {noformat} Caused by: org.apache.hadoop.ipc.RemoteException(java.lang.NullPointerException): java.lang.NullPointerException at org.apache.hadoop.yarn.server.resourcemanager.ResourceTrackerService.registerNodeManager(ResourceTrackerService.java:213) at org.apache.hadoop.yarn.server.api.impl.pb.service.ResourceTrackerPBServiceImpl.registerNodeManager(ResourceTrackerPBServiceImpl.java:54) {noformat}

• YARN-1816. Major sub-task reported by Arpit Gupta and fixed by Jian He
  Succeeded application remains in accepted after RM restart
  

  {code} 2014-03-10 18:07:31,944|beaver.machine|INFO|Application-Id Application-Name Application-Type User Queue State Final-State Progress Tracking-URL 2014-03-10 18:07:31,945|beaver.machine|INFO|application_1394449508064_0008 test_mapred_ha_multiple_job_nn-rm-1-min-5-jobs_1394449960-4 MAPREDUCE hrt_qa default ACCEPTED SUCCEEDED 100% http://hostname:19888/jobhistory/job/job_1394449508064_0008 2014-03-10 18:08:02,125|beaver.machine|INFO|RUNNING: /usr/bin/yarn application -list -appStates NEW,NEW_SAVING,SUBMITTED,ACCEPTED,RUNNING 2014-03-10 18:08:03,198|beaver.machine|INFO|14/03/10 18:08:03 INFO client.ConfiguredRMFailoverProxyProvider: Failing over to rm2 2014-03-10 18:08:03,238|beaver.machine|INFO|Total number of applications (application-types: [] and states: [NEW, NEW_SAVING, SUBMITTED, ACCEPTED, RUNNING]):1 2014-03-10 18:08:03,239|beaver.machine|INFO|Application-Id Application-Name Application-Type User Queue State Final-State Progress Tracking-URL 2014-03-10 18:08:03,239|beaver.machine|INFO|application_1394449508064_0008 test_mapred_ha_multiple_job_nn-rm-1-min-5-jobs_1394449960-4 MAPREDUCE hrt_qa default ACCEPTED SUCCEEDED 100% http://hostname:19888/jobhistory/job/job_1394449508064_0008 2014-03-10 18:08:33,390|beaver.machine|INFO|RUNNING: /usr/bin/yarn application -list -appStates NEW,NEW_SAVING,SUBMITTED,ACCEPTED,RUNNING 2014-03-10 18:08:34,437|beaver.machine|INFO|14/03/10 18:08:34 INFO client.ConfiguredRMFailoverProxyProvider: Failing over to rm2 2014-03-10 18:08:34,477|beaver.machine|INFO|Total number of applications (application-types: [] and states: [NEW, NEW_SAVING, SUBMITTED, ACCEPTED, RUNNING]):1 2014-03-10 18:08:34,477|beaver.machine|INFO|Application-Id Application-Name Application-Type User Queue State Final-State Progress Tracking-URL 2014-03-10 18:08:34,478|beaver.machine|INFO|application_1394449508064_0008 test_mapred_ha_multiple_job_nn-rm-1-min-5-jobs_1394449960-4 MAPREDUCE hrt_qa default ACCEPTED SUCCEEDED 100% http://hostname:19888/jobhistory/job/job_1394449508064_0008 2014-03-10 18:09:04,628|beaver.machine|INFO|RUNNING: /usr/bin/yarn application -list -appStates NEW,NEW_SAVING,SUBMITTED,ACCEPTED,RUNNING 2014-03-10 18:09:05,688|beaver.machine|INFO|14/03/10 18:09:05 INFO client.ConfiguredRMFailoverProxyProvider: Failing over to rm2 2014-03-10 18:09:05,728|beaver.machine|INFO|Total number of applications (application-types: [] and states: [NEW, NEW_SAVING, SUBMITTED, ACCEPTED, RUNNING]):1 2014-03-10 18:09:05,728|beaver.machine|INFO|Application-Id Application-Name Application-Type User Queue State Final-State Progress Tracking-URL 2014-03-10 18:09:05,729|beaver.machine|INFO|application_1394449508064_0008 test_mapred_ha_multiple_job_nn-rm-1-min-5-jobs_1394449960-4 MAPREDUCE hrt_qa default ACCEPTED SUCCEEDED 100% http://hostname:19888/jobhistory/job/job_1394449508064_0008 2014-03-10 18:09:35,879|beaver.machine|INFO|RUNNING: /usr/bin/yarn application -list -appStates NEW,NEW_SAVING,SUBMITTED,ACCEPTED,RUNNING 2014-03-10 18:09:36,951|beaver.machine|INFO|14/03/10 18:09:36 INFO client.ConfiguredRMFailoverProxyProvider: Failing over to rm2 2014-03-10 18:09:36,992|beaver.machine|INFO|Total number of applications (application-types: [] and states: [NEW, NEW_SAVING, SUBMITTED, ACCEPTED, RUNNING]):1 2014-03-10 18:09:36,993|beaver.machine|INFO|Application-Id Application-Name Application-Type User Queue State Final-State Progress Tracking-URL 2014-03-10 18:09:36,993|beaver.machine|INFO|application_1394449508064_0008 test_mapred_ha_multiple_job_nn-rm-1-min-5-jobs_1394449960-4 MAPREDUCE hrt_qa default ACCEPTED SUCCEEDED 100% http://hostname:19888/jobhistory/job/job_1394449508064_0008 2014-03-10 18:10:07,142|beaver.machine|INFO|RUNNING: /usr/bin/yarn application -list -appStates NEW,NEW_SAVING,SUBMITTED,ACCEPTED,RUNNING 2014-03-10 18:10:08,201|beaver.machine|INFO|14/03/10 18:10:08 INFO client.ConfiguredRMFailoverProxyProvider: Failing over to rm2 2014-03-10 18:10:08,242|beaver.machine|INFO|Total number of applications (application-types: [] and states: [NEW, NEW_SAVING, SUBMITTED, ACCEPTED, RUNNING]):1 2014-03-10 18:10:08,242|beaver.machine|INFO|Application-Id Application-Name Application-Type User Queue State Final-State Progress Tracking-URL 2014-03-10 18:10:08,242|beaver.machine|INFO|application_1394449508064_0008 test_mapred_ha_multiple_job_nn-rm-1-min-5-jobs_1394449960-4 MAPREDUCE hrt_qa default ACCEPTED SUCCEEDED 100% http://hostname:19888/jobhistory/job/job_1394449508064_0008 2014-03-10 18:10:38,392|beaver.machine|INFO|RUNNING: /usr/bin/yarn application -list -appStates NEW,NEW_SAVING,SUBMITTED,ACCEPTED,RUNNING 2014-03-10 18:10:39,443|beaver.machine|INFO|14/03/10 18:10:39 INFO client.ConfiguredRMFailoverProxyProvider: Failing over to rm2 2014-03-10 18:10:39,484|beaver.machine|INFO|Total number of applications (application-types: [] and states: [NEW, NEW_SAVING, SUBMITTED, ACCEPTED, RUNNING]):1 2014-03-10 18:10:39,484|beaver.machine|INFO|Application-Id Application-Name Application-Type User Queue State Final-State Progress Tracking-URL 2014-03-10 18:10:39,485|beaver.machine|INFO|application_1394449508064_0008 test_mapred_ha_multiple_job_nn-rm-1-min-5-jobs_1394449960-4 MAPREDUCE hrt_qa default ACCEPTED SUCCEEDED 100% http://hostname:19888/jobhistory/job/job_1394449508064_0008 {code}

• YARN-1812. Major sub-task reported by Yesha Vora and fixed by Jian He
  Job stays in PREP state for long time after RM Restarts
  

  Steps followed: 1) start a sort job with 80 maps and 5 reducers 2) restart Resource manager when 60 maps and 0 reducers are finished 3) Wait for job to come out of PREP state. The job does not come out of PREP state after 7-8 mins. After waiting for 7-8 mins, test kills the job. However, Sort job should not take this long time to come out of PREP state

• YARN-1811. Major sub-task reported by Robert Kanter and fixed by Robert Kanter (resourcemanager)
  RM HA: AM link broken if the AM is on nodes other than RM
  

  When using RM HA, if you click on the "Application Master" link in the RM web UI while the job is running, you get an Error 500:

• YARN-1800. Critical sub-task reported by Paul Isaychuk and fixed by Varun Vasudev (nodemanager)
  YARN NodeManager with java.util.concurrent.RejectedExecutionException
  

  Noticed this on tests running on Apache Hadoop 2.2 cluster {code} 2014-01-23 01:30:28,575 INFO localizer.LocalizedResource (LocalizedResource.java:handle(196)) - Resource hdfs://colo-2:8020/user/fertrist/oozie-oozi/0000605-140114233013619-oozie-oozi-W/aggregator--map-reduce/map-reduce-launcher.jar transitioned from INIT to DOWNLOADING 2014-01-23 01:30:28,575 INFO localizer.LocalizedResource (LocalizedResource.java:handle(196)) - Resource hdfs://colo-2:8020/user/fertrist/.staging/job_1389742077466_0396/job.splitmetainfo transitioned from INIT to DOWNLOADING 2014-01-23 01:30:28,575 INFO localizer.LocalizedResource (LocalizedResource.java:handle(196)) - Resource hdfs://colo-2:8020/user/fertrist/.staging/job_1389742077466_0396/job.split transitioned from INIT to DOWNLOADING 2014-01-23 01:30:28,575 INFO localizer.LocalizedResource (LocalizedResource.java:handle(196)) - Resource hdfs://colo-2:8020/user/fertrist/.staging/job_1389742077466_0396/job.xml transitioned from INIT to DOWNLOADING 2014-01-23 01:30:28,576 INFO localizer.ResourceLocalizationService (ResourceLocalizationService.java:addResource(651)) - Downloading public rsrc:{ hdfs://colo-2:8020/user/fertrist/oozie-oozi/0000605-140114233013619-oozie-oozi-W/aggregator--map-reduce/map-reduce-launcher.jar, 1390440627435, FILE, null } 2014-01-23 01:30:28,576 FATAL event.AsyncDispatcher (AsyncDispatcher.java:dispatch(141)) - Error in dispatcher thread java.util.concurrent.RejectedExecutionException at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:1768) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:767) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:658) at java.util.concurrent.ExecutorCompletionService.submit(ExecutorCompletionService.java:152) at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$PublicLocalizer.addResource(ResourceLocalizationService.java:678) at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerTracker.handle(ResourceLocalizationService.java:583) at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerTracker.handle(ResourceLocalizationService.java:525) at org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:134) at org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:81) at java.lang.Thread.run(Thread.java:662) 2014-01-23 01:30:28,577 INFO event.AsyncDispatcher (AsyncDispatcher.java:dispatch(144)) - Exiting, bbye.. 2014-01-23 01:30:28,596 INFO mortbay.log (Slf4jLog.java:info(67)) - Stopped SelectChannelConnector@0.0.0.0:50060 2014-01-23 01:30:28,597 INFO containermanager.ContainerManagerImpl (ContainerManagerImpl.java:cleanUpApplicationsOnNMShutDown(328)) - Applications still running : [application_1389742077466_0396] 2014-01-23 01:30:28,597 INFO containermanager.ContainerManagerImpl (ContainerManagerImpl.java:cleanUpApplicationsOnNMShutDown(336)) - Wa {code}

• YARN-1793. Critical bug reported by Karthik Kambatla and fixed by Karthik Kambatla (resourcemanager)
  yarn application -kill doesn't kill UnmanagedAMs
  

  Trying to kill an Unmanaged AM though CLI (yarn application -kill <id>) logs a success, but doesn't actually kill the AM or reclaim the containers allocated to it.

• YARN-1789. Minor improvement reported by Akira AJISAKA and fixed by Tsuyoshi OZAWA (resourcemanager)
  ApplicationSummary does not escape newlines in the app name
  

  YARN-side of MAPREDUCE-5778. ApplicationSummary is not escaping newlines in the app name. This can result in an application summary log entry that spans multiple lines when users are expecting one-app-per-line output.

• YARN-1788. Critical bug reported by Tassapol Athiapinya and fixed by Varun Vasudev (resourcemanager)
  AppsCompleted/AppsKilled metric is incorrect when MR job is killed with yarn application -kill
  

  Run MR sleep job. Kill the application in RUNNING state. Observe RM metrics. Expecting AppsCompleted = 0/AppsKilled = 1 Actual is AppsCompleted = 1/AppsKilled = 0

• YARN-1787. Major sub-task reported by Zhijie Shen and fixed by Zhijie Shen
  yarn applicationattempt/container print wrong usage information
  

  yarn applicationattempt prints: {code} Invalid Command Usage : usage: application -appStates <States> Works with -list to filter applications based on input comma-separated list of application states. The valid application state can be one of the following: ALL,NEW,NEW_SAVING,SUBMITTED,ACCEPTED,RUN NING,FINISHED,FAILED,KILLED -appTypes <Types> Works with -list to filter applications based on input comma-separated list of application types. -help Displays help for all commands. -kill <Application ID> Kills the application. -list <arg> List application attempts for aplication from AHS. -movetoqueue <Application ID> Moves the application to a different queue. -queue <Queue Name> Works with the movetoqueue command to specify which queue to move an application to. -status <Application ID> Prints the status of the application. {code} yarn container prints: {code} Invalid Command Usage : usage: application -appStates <States> Works with -list to filter applications based on input comma-separated list of application states. The valid application state can be one of the following: ALL,NEW,NEW_SAVING,SUBMITTED,ACCEPTED,RUN NING,FINISHED,FAILED,KILLED -appTypes <Types> Works with -list to filter applications based on input comma-separated list of application types. -help Displays help for all commands. -kill <Application ID> Kills the application. -list <arg> List application attempts for aplication from AHS. -movetoqueue <Application ID> Moves the application to a different queue. -queue <Queue Name> Works with the movetoqueue command to specify which queue to move an application to. -status <Application ID> Prints the status of the application. {code} Both commands print irrelevant yarn application usage information.

• YARN-1785. Major bug reported by bc Wong and fixed by bc Wong
  FairScheduler treats app lookup failures as ERRORs
  

  When invoking the /ws/v1/cluster/apps endpoint, RM will eventually get to RMAppImpl#createAndGetApplicationReport, which calls RMAppAttemptImpl#getApplicationResourceUsageReport, which looks up the app in the scheduler, which may or may not exist. So FairScheduler shouldn't log an error for every lookup failure: {noformat} 2014-02-17 08:23:21,240 ERROR org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler: Request for appInfo of unknown attemptappattempt_1392419715319_0135_000001 {noformat}

• YARN-1783. Critical bug reported by Arpit Gupta and fixed by Jian He
  yarn application does not make any progress even when no other application is running when RM is being restarted in the background
  

  Noticed that during HA tests some tests took over 3 hours to run when the test failed. Looking at the logs i see the application made no progress for a very long time. However if i look at application log from yarn it actually ran in 5 mins I am seeing same behavior when RM was being restarted in the background and when both RM and AM were being restarted. This does not happen for all applications but a few will hit this in the nightly run.

• YARN-1781. Major sub-task reported by Varun Vasudev and fixed by Varun Vasudev (nodemanager)
  NM should allow users to specify max disk utilization for local disks
  

  This is related to YARN-257(it's probably a sub task?). Currently, the NM does not detect full disks and allows full disks to be used by containers leading to repeated failures. YARN-257 deals with graceful handling of full disks. This ticket is only about detection of full disks by the disk health checkers. The NM should allow users to set a maximum disk utilization for local disks and mark disks as bad once they exceed that utilization. At the very least, the NM should at least detect full disks.

• YARN-1780. Major sub-task reported by Zhijie Shen and fixed by Zhijie Shen
  Improve logging in timeline service
  

  It's difficult to trace whether the client has successfully posted the entity to the timeline service or not.

• YARN-1776. Major sub-task reported by Zhijie Shen and fixed by Zhijie Shen
  renewDelegationToken should survive RM failover
  

  When a delegation token is renewed, two RMStateStore operations: 1) removing the old DT, and 2) storing the new DT will happen. If RM fails in between. There would be problem.

• YARN-1775. Major sub-task reported by Rajesh Balamohan and fixed by Rajesh Balamohan (nodemanager)
  Create SMAPBasedProcessTree to get PSS information
  

  Create SMAPBasedProcessTree (by extending ProcfsBasedProcessTree), which will make use of PSS for computing the memory usage.

• YARN-1774. Blocker bug reported by Anubhav Dhoot and fixed by Anubhav Dhoot (resourcemanager)
  FS: Submitting to non-leaf queue throws NPE
  

  If you create a hierarchy of queues and assign a job to parent queue, FairScheduler quits with a NPE.

• YARN-1771. Critical improvement reported by Sangjin Lee and fixed by Sangjin Lee (nodemanager)
  many getFileStatus calls made from node manager for localizing a public distributed cache resource
  

  We're observing that the getFileStatus calls are putting a fair amount of load on the name node as part of checking the public-ness for localizing a resource that belong in the public cache. We see 7 getFileStatus calls made for each of these resource. We should look into reducing the number of calls to the name node. One example: {noformat} 2014-02-27 18:07:27,351 INFO audit: ... cmd=getfileinfo src=/tmp/temp-887708724/tmp883330348/foo-0.0.44.jar ... 2014-02-27 18:07:27,352 INFO audit: ... cmd=getfileinfo src=/tmp/temp-887708724/tmp883330348/foo-0.0.44.jar ... 2014-02-27 18:07:27,352 INFO audit: ... cmd=getfileinfo src=/tmp/temp-887708724/tmp883330348 ... 2014-02-27 18:07:27,353 INFO audit: ... cmd=getfileinfo src=/tmp/temp-887708724 ... 2014-02-27 18:07:27,353 INFO audit: ... cmd=getfileinfo src=/tmp ... 2014-02-27 18:07:27,354 INFO audit: ... cmd=getfileinfo src=/ ... 2014-02-27 18:07:27,354 INFO audit: ... cmd=getfileinfo src=/tmp/temp-887708724/tmp883330348/foo-0.0.44.jar ... 2014-02-27 18:07:27,355 INFO audit: ... cmd=open src=/tmp/temp-887708724/tmp883330348/foo-0.0.44.jar ... {noformat}

• YARN-1768. Minor bug reported by Hitesh Shah and fixed by Tsuyoshi OZAWA (client)
  yarn kill non-existent application is too verbose
  

  Instead of catching ApplicationNotFound and logging a simple app not found message, the whole stack trace is logged.

• YARN-1766. Major sub-task reported by Xuan Gong and fixed by Xuan Gong
  When RM does the initiation, it should use loaded Configuration instead of bootstrap configuration.
  

  Right now, we have FileSystemBasedConfigurationProvider to let Users upload the configurations into remote File System, and let different RMs share the same configurations. During the initiation, RM will load the configurations from Remote File System. So when RM initiates the services, it should use the loaded Configurations instead of using the bootstrap configurations.

• YARN-1765. Major sub-task reported by Xuan Gong and fixed by Xuan Gong
  Write test cases to verify that killApplication API works in RM HA
  

• YARN-1764. Major sub-task reported by Xuan Gong and fixed by Xuan Gong
  Handle RM fail overs after the submitApplication call.
  

• YARN-1761. Major sub-task reported by Xuan Gong and fixed by Xuan Gong
  RMAdminCLI should check whether HA is enabled before executes transitionToActive/transitionToStandby
  

• YARN-1760. Trivial bug reported by Karthik Kambatla and fixed by Karthik Kambatla
  TestRMAdminService assumes CapacityScheduler
  

  YARN-1611 adds TestRMAdminService which assumes the use of CapacityScheduler. {noformat} java.lang.ClassCastException: org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler cannot be cast to org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler at org.apache.hadoop.yarn.server.resourcemanager.TestRMAdminService.testAdminRefreshQueuesWithFileSystemBasedConfigurationProvider(TestRMAdminService.java:115) {noformat}

• YARN-1758. Blocker bug reported by Hitesh Shah and fixed by Xuan Gong
  MiniYARNCluster broken post YARN-1666
  

  NPE seen when trying to use MiniYARNCluster

• YARN-1752. Major bug reported by Jian He and fixed by Rohith
  Unexpected Unregistered event at Attempt Launched state
  

  {code} 2014-02-21 14:56:03,453 ERROR org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl: Can't handle this event at current state org.apache.hadoop.yarn.state.InvalidStateTransitonException: Invalid event: UNREGISTERED at LAUNCHED at org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:305) at org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:46) at org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:448) at org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl.handle(RMAppAttemptImpl.java:647) at org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl.handle(RMAppAttemptImpl.java:103) at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$ApplicationAttemptEventDispatcher.handle(ResourceManager.java:733) at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$ApplicationAttemptEventDispatcher.handle(ResourceManager.java:714) at org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:173) at org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:106) at java.lang.Thread.run(Thread.java:695) {code}

• YARN-1749. Major sub-task reported by Zhijie Shen and fixed by Zhijie Shen
  Review AHS configs and sync them up with the timeline-service configs
  

  We need to: 1. Review the configuration names and default values 2. Combine the two store class configurations Some other thoughts: 1. Maybe we don't need null implementation of ApplicationHistoryStore any more 2. Maybe if yarn.ahs.enabled = false, we should stop AHS web server returning historic information

• YARN-1748. Blocker bug reported by Sravya Tirukkovalur and fixed by Sravya Tirukkovalur
  hadoop-yarn-server-tests packages core-site.xml breaking downstream tests
  

  Jars should not package config files, as this might come into the classpaths of clients causing the clients to break.

• YARN-1742. Trivial bug reported by Akira AJISAKA and fixed by Akira AJISAKA (documentation)
  Fix javadoc of parameter DEFAULT_NM_MIN_HEALTHY_DISKS_FRACTION
  

  In YarnConfiguration.java, {code} /** * By default, at least 5% of disks are to be healthy to say that the node * is healthy in terms of disks. */ public static final float DEFAULT_NM_MIN_HEALTHY_DISKS_FRACTION = 0.25F; {code} 25% is the correct.

• YARN-1734. Critical sub-task reported by Xuan Gong and fixed by Xuan Gong
  RM should get the updated Configurations when it transits from Standby to Active
  

  Currently, we have ConfigurationProvider which can support LocalConfiguration, and FileSystemBasedConfiguration. When HA is enabled, and FileSystemBasedConfiguration is enabled, RM can not get the updated Configurations when it transits from Standby to Active

• YARN-1732. Major sub-task reported by Billie Rinaldi and fixed by Billie Rinaldi
  Change types of related entities and primary filters in ATSEntity
  

  The current types Map<String, List<String>> relatedEntities and Map<String, Object> primaryFilters have issues. The List<String> value of the related entities map could have multiple identical strings in it, which doesn't make sense. A more major issue is that we cannot allow primary filter values to be overwritten, because otherwise we will be unable to find those primary filter entries when we want to delete an entity (without doing a nearly full scan). I propose changing related entities to Map<String, Set<String>> and primary filters to Map<String, Set<Object>>. The basic methods to add primary filters and related entities are of the form add(key, value) and will not need to change.

• YARN-1730. Major sub-task reported by Billie Rinaldi and fixed by Billie Rinaldi
  Leveldb timeline store needs simple write locking
  

  Although the leveldb writes are performed atomically in a batch, a start time for the entity needs to identified before each write. Thus a per-entity write lock should be acquired.

• YARN-1729. Major sub-task reported by Billie Rinaldi and fixed by Billie Rinaldi
  TimelineWebServices always passes primary and secondary filters as strings
  

  Primary filters and secondary filter values can be arbitrary json-compatible Object. The web services should determine if the filters specified as query parameters are objects or strings before passing them to the store.

• YARN-1724. Critical bug reported by Sandy Ryza and fixed by Sandy Ryza (scheduler)
  Race condition in Fair Scheduler when continuous scheduling is turned on
  

  If nodes resource allocations change during Collections.sort(nodeIdList, nodeAvailableResourceComparator); we'll hit: java.lang.IllegalArgumentException: Comparison method violates its general contract!

• YARN-1721. Critical bug reported by Sandy Ryza and fixed by Sandy Ryza (scheduler)
  When moving app between queues in Fair Scheduler, grab lock on FSSchedulerApp
  

  FairScheduler.moveApplication should grab lock on FSSchedulerApp, so that allocate() can't be modifying it at the same time.

• YARN-1719. Major sub-task reported by Billie Rinaldi and fixed by Billie Rinaldi
  ATSWebServices produces jersey warnings
  

  These don't appear to affect how the web services work, but the following warnings are logged: {noformat} WARNING: The following warnings have been detected with resource and/or provider classes: WARNING: A sub-resource method, public org.apache.hadoop.yarn.server.applicati onhistoryservice.webapp.ATSWebServices$AboutInfo org.apache.hadoop.yarn.server.a pplicationhistoryservice.webapp.ATSWebServices.about(javax.servlet.http.HttpServ letRequest,javax.servlet.http.HttpServletResponse), with URI template, "/", is t reated as a resource method WARNING: A sub-resource method, public org.apache.hadoop.yarn.api.records.appt imeline.ATSPutErrors org.apache.hadoop.yarn.server.applicationhistoryservice.web app.ATSWebServices.postEntities(javax.servlet.http.HttpServletRequest,javax.serv let.http.HttpServletResponse,org.apache.hadoop.yarn.api.records.apptimeline.ATSE ntities), with URI template, "/", is treated as a resource method {noformat}

• YARN-1717. Major sub-task reported by Billie Rinaldi and fixed by Billie Rinaldi
  Enable offline deletion of entries in leveldb timeline store
  

  The leveldb timeline store implementation needs the following: * better documentation of its internal structures * internal changes to enable deleting entities ** never overwrite existing primary filter entries ** add hidden reverse pointers to related entities

• YARN-1706. Major sub-task reported by Zhijie Shen and fixed by Zhijie Shen
  Create an utility function to dump timeline records to json
  

  For verification and log purpose

• YARN-1704. Blocker sub-task reported by Billie Rinaldi and fixed by Billie Rinaldi
  Review LICENSE and NOTICE to reflect new levelDB releated libraries being used
  

  Make any changes necessary in LICENSE and NOTICE related to dependencies introduced by the application timeline store.

• YARN-1698. Major sub-task reported by Zhijie Shen and fixed by Zhijie Shen
  Replace MemoryApplicationTimelineStore with LeveldbApplicationTimelineStore as default
  

• YARN-1697. Major bug reported by Sandy Ryza and fixed by Sandy Ryza (nodemanager)
  NodeManager reports negative running containers
  

  We're seeing the NodeManager metrics report a negative number of running containers.

• YARN-1692. Major bug reported by Sangjin Lee and fixed by Sangjin Lee (scheduler)
  ConcurrentModificationException in fair scheduler AppSchedulable
  

  We saw a ConcurrentModificationException thrown in the fair scheduler: {noformat} 2014-02-07 01:40:01,978 ERROR org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler: Exception in fair scheduler UpdateThread java.util.ConcurrentModificationException at java.util.HashMap$HashIterator.nextEntry(HashMap.java:926) at java.util.HashMap$ValueIterator.next(HashMap.java:954) at org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.AppSchedulable.updateDemand(AppSchedulable.java:85) at org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSLeafQueue.updateDemand(FSLeafQueue.java:125) at org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSParentQueue.updateDemand(FSParentQueue.java:82) at org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler.update(FairScheduler.java:217) at org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler$UpdateThread.run(FairScheduler.java:195) at java.lang.Thread.run(Thread.java:724) {noformat} The map that gets returned by FSSchedulerApp.getResourceRequests() are iterated on without proper synchronization.

• YARN-1690. Major sub-task reported by Mayank Bansal and fixed by Mayank Bansal
  Sending timeline entities+events from Distributed shell
  

• YARN-1689. Critical bug reported by Deepesh Khandelwal and fixed by Vinod Kumar Vavilapalli (resourcemanager)
  RMAppAttempt is not killed when RMApp is at ACCEPTED
  

  When running some Hive on Tez jobs, the RM after a while gets into an unusable state where no jobs run. In the RM log I see the following exception: {code} 2014-02-04 20:28:08,553 WARN ipc.Server (Server.java:run(1978)) - IPC Server handler 0 on 8030, call org.apache.hadoop.yarn.api.ApplicationMasterProtocolPB.registerApplicationMaster from 172.18.145.156:40474 Call#0 Retry#0: error: java.lang.NullPointerException java.lang.NullPointerException at org.apache.hadoop.yarn.server.resourcemanager.scheduler.AbstractYarnScheduler.getTransferredContainers(AbstractYarnScheduler.java:48) at org.apache.hadoop.yarn.server.resourcemanager.ApplicationMasterService.registerApplicationMaster(ApplicationMasterService.java:278) at org.apache.hadoop.yarn.api.impl.pb.service.ApplicationMasterProtocolPBServiceImpl.registerApplicationMaster(ApplicationMasterProtocolPBServiceImpl.java:90) at org.apache.hadoop.yarn.proto.ApplicationMasterProtocol$ApplicationMasterProtocolService$2.callBlockingMethod(ApplicationMasterProtocol.java:95) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:585) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:928) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1962) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1958) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1956) ...... 2014-02-04 20:28:08,544 ERROR rmapp.RMAppImpl (RMAppImpl.java:handle(626)) - Can't handle this event at current state org.apache.hadoop.yarn.state.InvalidStateTransitonException: Invalid event: ATTEMPT_REGISTERED at KILLED at org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:305) at org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:46) at org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:448) at org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl.handle(RMAppImpl.java:624) at org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl.handle(RMAppImpl.java:81) at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$ApplicationEventDispatcher.handle(ResourceManager.java:656) at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$ApplicationEventDispatcher.handle(ResourceManager.java:640) at org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:173) at org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:106) at java.lang.Thread.run(Thread.java:662) 2014-02-04 20:28:08,549 INFO resourcemanager.RMAuditLogger (RMAuditLogger.java:logSuccess(140)) - USER=hrt_qa IP=172.18.145.156 OPERATION=Kill Application Request TARGET=ClientRMService RESULT=SUCCESS APPID=application_1391543307203_0001 2014-02-04 20:28:08,553 WARN ipc.Server (Server.java:run(1978)) - IPC Server handler 0 on 8030, call org.apache.hadoop.yarn.api.ApplicationMasterProtocolPB.registerApplicationMaster from 172.18.145.156:40474 Call#0 Retry#0: error: java.lang.NullPointerException java.lang.NullPointerException at org.apache.hadoop.yarn.server.resourcemanager.scheduler.AbstractYarnScheduler.getTransferredContainers(AbstractYarnScheduler.java:48) at org.apache.hadoop.yarn.server.resourcemanager.ApplicationMasterService.registerApplicationMaster(ApplicationMasterService.java:278) at org.apache.hadoop.yarn.api.impl.pb.service.ApplicationMasterProtocolPBServiceImpl.registerApplicationMaster(ApplicationMasterProtocolPBServiceImpl.java:90) at org.apache.hadoop.yarn.proto.ApplicationMasterProtocol$ApplicationMasterProtocolService$2.callBlockingMethod(ApplicationMasterProtocol.java:95) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:585) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:928) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1962) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1958) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1956) {code}

• YARN-1687. Major sub-task reported by Zhijie Shen and fixed by Zhijie Shen
  Refactoring timeline classes to remove "app" related words
  

  Remove ATS prefix, change package name, fix javadoc and so on

• YARN-1686. Major bug reported by Rohith and fixed by Rohith (nodemanager)
  NodeManager.resyncWithRM() does not handle exception which cause NodeManger to Hang.
  

  During start of NodeManager,if registration with resourcemanager throw exception then nodemager shutdown happens. Consider case where NM-1 is registered with RM. RM issued Resync to NM. If any exception thrown in "resyncWithRM" (starts new thread which does not handle exception) during RESYNC evet, then this thread is lost. NodeManger enters hanged state.

• YARN-1685. Major sub-task reported by Mayank Bansal and fixed by Zhijie Shen
  Bugs around log URL
  

  1. Log URL should be different when the container is running and finished 2. Null case needs to be handled 3. The way of constructing log URL should be corrected

• YARN-1684. Major sub-task reported by Billie Rinaldi and fixed by Billie Rinaldi
  Fix history server heap size in yarn script
  

  The yarn script currently has the following: {noformat} if [ "$YARN_RESOURCEMANAGER_HEAPSIZE" != "" ]; then JAVA_HEAP_MAX="-Xmx""$YARN_HISTORYSERVER_HEAPSIZE""m" fi {noformat}

• YARN-1676. Major sub-task reported by Xuan Gong and fixed by Xuan Gong
  Make admin refreshUserToGroupsMappings of configuration work across RM failover
  

• YARN-1673. Blocker bug reported by Tassapol Athiapinya and fixed by Mayank Bansal (client)
  Valid yarn kill application prints out help message.
  

  yarn application -kill <application ID> used to work previously. In 2.4.0 it prints out help message and does not kill the application.

• YARN-1672. Trivial bug reported by Karthik Kambatla and fixed by Naren Koneru (nodemanager)
  YarnConfiguration is missing a default for yarn.nodemanager.log.retain-seconds
  

  YarnConfiguration is missing a default for yarn.nodemanager.log.retain-seconds

• YARN-1670. Critical bug reported by Thomas Graves and fixed by Mit Desai
  aggregated log writer can write more log data then it says is the log length
  

  We have seen exceptions when using 'yarn logs' to read log files. at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65) at java.lang.Long.parseLong(Long.java:441) at java.lang.Long.parseLong(Long.java:483) at org.apache.hadoop.yarn.logaggregation.AggregatedLogFormat$LogReader.readAContainerLogsForALogType(AggregatedLogFormat.java:518) at org.apache.hadoop.yarn.logaggregation.LogDumper.dumpAContainerLogs(LogDumper.java:178) at org.apache.hadoop.yarn.logaggregation.LogDumper.run(LogDumper.java:130) at org.apache.hadoop.yarn.logaggregation.LogDumper.main(LogDumper.java:246) We traced it down to the reader trying to read the file type of the next file but where it reads is still log data from the previous file. What happened was the Log Length was written as a certain size but the log data was actually longer then that. Inside of the write() routine in LogValue it first writes what the logfile length is, but then when it goes to write the log itself it just goes to the end of the file. There is a race condition here where if someone is still writing to the file when it goes to be aggregated the length written could be to small. We should have the write() routine stop when it writes whatever it said was the length. It would be nice if we could somehow tell the user it might be truncated but I'm not sure of a good way to do this. We also noticed that a bug in readAContainerLogsForALogType where it is using an int for curRead whereas it should be using a long. while (len != -1 && curRead < fileLength) { This isn't actually a problem right now as it looks like the underlying decoder is doing the right thing and the len condition exits.

• YARN-1669. Major sub-task reported by Xuan Gong and fixed by Xuan Gong
  Make admin refreshServiceAcls work across RM failover
  

• YARN-1668. Major sub-task reported by Xuan Gong and fixed by Xuan Gong
  Make admin refreshAdminAcls work across RM failover
  

  Change the handling of admin-acls to be available across RM failover by making using of a remote configuration-provider

• YARN-1667. Major sub-task reported by Xuan Gong and fixed by Xuan Gong
  Make admin refreshSuperUserGroupsConfiguration work across RM failover
  

• YARN-1666. Major sub-task reported by Xuan Gong and fixed by Xuan Gong
  Make admin refreshNodes work across RM failover
  

• YARN-1665. Major sub-task reported by Arpit Gupta and fixed by Xuan Gong (resourcemanager)
  Set better defaults for HA configs for automatic failover
  

  In order to enable HA (automatic failover) i had to set the following configs {code} <property> <name>yarn.resourcemanager.ha.enabled</name> <value>true</value> </property> <property> <name>yarn.resourcemanager.ha.automatic-failover.enabled</name> <value>true</value> </property> <property> <name>yarn.resourcemanager.ha.automatic-failover.embedded</name> <value>true</value> </property> {code} I believe the user should just have to set yarn.resourcemanager.ha.enabled=true and the rest should be set as defaults. Basically automatic failover should be the default.

• YARN-1661. Major bug reported by Tassapol Athiapinya and fixed by Vinod Kumar Vavilapalli (applications/distributed-shell)
  AppMaster logs says failing even if an application does succeed.
  

  Run: /usr/bin/yarn org.apache.hadoop.yarn.applications.distributedshell.Client -jar <distributed shell jar> -shell_command ls Open AM logs. Last line would indicate AM failure even though container logs print good ls result. {code} 2014-01-24 21:45:29,592 INFO [main] distributedshell.ApplicationMaster (ApplicationMaster.java:finish(599)) - Application completed. Signalling finish to RM 2014-01-24 21:45:29,612 INFO [main] impl.AMRMClientImpl (AMRMClientImpl.java:unregisterApplicationMaster(315)) - Waiting for application to be successfully unregistered. 2014-01-24 21:45:29,816 INFO [main] distributedshell.ApplicationMaster (ApplicationMaster.java:main(267)) - Application Master failed. exiting {code}

• YARN-1660. Major sub-task reported by Arpit Gupta and fixed by Xuan Gong (resourcemanager)
  add the ability to set yarn.resourcemanager.hostname.rm-id instead of setting all the various host:port properties for RM
  

  Currently the user has to specify all the various host:port properties for RM. We should follow the pattern that we do for non HA setup where we can specify yarn.resourcemanager.hostname.rm-id and the defaults are used for all other affected properties.

• YARN-1659. Major sub-task reported by Billie Rinaldi and fixed by Billie Rinaldi
  Define the ApplicationTimelineStore store as an abstraction for implementing different storage impls for storing timeline information
  

  These will be used by ApplicationTimelineStore interface. The web services will convert the store-facing obects to the user-facing objects.

• YARN-1658. Major sub-task reported by Cindy Li and fixed by Cindy Li
  Webservice should redirect to active RM when HA is enabled.
  

  When HA is enabled, web service to standby RM should be redirected to the active RM. This is a related Jira to YARN-1525.

• YARN-1641. Major sub-task reported by Karthik Kambatla and fixed by Karthik Kambatla (resourcemanager)
  ZK store should attempt a write periodically to ensure it is still Active
  

  Fencing in ZK store kicks in when the RM tries to write something to the store. If the RM doesn't write anything to the store, it doesn't get fenced and can continue to assume being the Active. By periodically writing a file (say, every RM_ZK_TIMEOUT_MS seconds), we can ensure it gets fenced.

• YARN-1640. Blocker sub-task reported by Xuan Gong and fixed by Xuan Gong
  Manual Failover does not work in secure clusters
  

  NodeManager gets rejected after manually making one RM as active.

• YARN-1639. Major sub-task reported by Arpit Gupta and fixed by Xuan Gong (resourcemanager)
  YARM RM HA requires different configs on different RM hosts
  

  We need to set yarn.resourcemanager.ha.id to rm1 or rm2 based on which rm you want to first or second. This means we have different configs on different RM nodes. This is unlike HDFS HA where the same configs are pushed to both NN's and it would be better to have the same setup for RM as this would make installation and managing easier.

• YARN-1637. Major sub-task reported by Vinod Kumar Vavilapalli and fixed by Zhijie Shen
  Implement a client library for java users to post entities+events
  

  This is a wrapper around the web-service to facilitate easy posting of entity+event data to the time-line server.

• YARN-1636. Major sub-task reported by Vinod Kumar Vavilapalli and fixed by Zhijie Shen
  Implement timeline related web-services inside AHS for storing and retrieving entities+events
  

• YARN-1635. Major sub-task reported by Vinod Kumar Vavilapalli and fixed by Billie Rinaldi
  Implement a Leveldb based ApplicationTimelineStore
  

  As per the design doc, we need a levelDB + local-filesystem based implementation to start with and for small deployments.

• YARN-1634. Major sub-task reported by Vinod Kumar Vavilapalli and fixed by Zhijie Shen
  Define an in-memory implementation of ApplicationTimelineStore
  

  As per the design doc, the store needs to pluggable. We need a base interface, and an in-memory implementation for testing.

• YARN-1633. Major sub-task reported by Vinod Kumar Vavilapalli and fixed by Zhijie Shen
  Define user-faced entity, entity-info and event objects
  

  Define the core objects of the application-timeline effort.

• YARN-1632. Minor bug reported by Chen He and fixed by Chen He
  TestApplicationMasterServices should be under org.apache.hadoop.yarn.server.resourcemanager package
  

  ApplicationMasterService is under org.apache.hadoop.yarn.server.resourcemanager package. However, its unit test file TestApplicationMasterService is placed under org.apache.hadoop.yarn.server.resourcemanager.applicationmasterservice package which only contains one file (TestApplicationMasterService).

• YARN-1625. Trivial sub-task reported by Shinichi Yamashita and fixed by Shinichi Yamashita
  mvn apache-rat:check outputs warning message in YARN-321 branch
  

  When I ran dev-support/test-patch.sh, following message output. {code} mvn apache-rat:check -DHadoopPatchProcess > /tmp/patchReleaseAuditOutput.txt 2>&1 There appear to be 1 release audit warnings after applying the patch. {code} {code} !????? /home/sinchii/git/YARN-321-test/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/webapps/applicationhistory/.keep Lines that start with ????? in the release audit report indicate files that do not have an Apache license header. {code} To avoid release audit warning, it should fix pom.xml.

• YARN-1617. Major bug reported by Sandy Ryza and fixed by Sandy Ryza (scheduler)
  Remove ancient comment and surround LOG.debug in AppSchedulingInfo.allocate
  

  {code} synchronized private void allocate(Container container) { // Update consumption and track allocations //TODO: fixme sharad /* try { store.storeContainer(container); } catch (IOException ie) { // TODO fix this. we shouldnt ignore }*/ LOG.debug("allocate: applicationId=" + applicationId + " container=" + container.getId() + " host=" + container.getNodeId().toString()); } {code}

• YARN-1613. Major sub-task reported by Zhijie Shen and fixed by Akira AJISAKA
  Fix config name YARN_HISTORY_SERVICE_ENABLED
  

  YARN_HISTORY_SERVICE_ENABLED property name is "yarn.ahs..enabled", which is wrong.

• YARN-1611. Major sub-task reported by Xuan Gong and fixed by Xuan Gong
  Make admin refresh of capacity scheduler configuration work across RM failover
  

  Currently, If we do refresh* for a standby RM, it will failover to the current active RM, and do the refresh* based on the local configuration file of the active RM.

• YARN-1605. Major sub-task reported by Vinod Kumar Vavilapalli and fixed by Vinod Kumar Vavilapalli
  Fix formatting issues with new module in YARN-321 branch
  

  There are a bunch of formatting issues. I'm restricting myself for a sweep of all the files in the new module.

• YARN-1597. Major sub-task reported by Vinod Kumar Vavilapalli and fixed by Vinod Kumar Vavilapalli
  FindBugs warnings on YARN-321 branch
  

  There are a bunch of findBugs warnings on YARN-321 branch.

• YARN-1596. Major sub-task reported by Vinod Kumar Vavilapalli and fixed by Vinod Kumar Vavilapalli
  Javadoc failures on YARN-321 branch
  

  There are some javadoc issues on YARN-321 branch.

• YARN-1595. Major sub-task reported by Vinod Kumar Vavilapalli and fixed by Vinod Kumar Vavilapalli
  Test failures on YARN-321 branch
  

  mvn test doesn't pass on YARN-321 branch anymore.

• YARN-1594. Major sub-task reported by Vinod Kumar Vavilapalli and fixed by Vinod Kumar Vavilapalli
  YARN-321 branch needs to be updated after YARN-888 pom changes
  

  YARN-888 changed the pom structure. And so latest merge to trunk breaks YARN-321 branch.

• YARN-1591. Major bug reported by Vinod Kumar Vavilapalli and fixed by Tsuyoshi OZAWA
  TestResourceTrackerService fails randomly on trunk
  

  As evidenced by Jenkins at https://issues.apache.org/jira/browse/YARN-1041?focusedCommentId=13868621&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13868621. It's failing randomly on trunk on my local box too

• YARN-1590. Major bug reported by Mohammad Kamrul Islam and fixed by Mohammad Kamrul Islam (resourcemanager)
  _HOST doesn't expand properly for RM, NM, ProxyServer and JHS
  

  _HOST is not properly substituted when we use VIP address. Currently it always used the host name of the machine and disregard the VIP address. It is true mainly for RM, NM, WebProxy, and JHS rpc service. Looks like it is working fine for webservice authentication. On the other hand, the same thing is working fine for NN and SNN in RPC as well as webservice.

• YARN-1588. Major sub-task reported by Jian He and fixed by Jian He
  Rebind NM tokens for previous attempt's running containers to the new attempt
  

• YARN-1587. Major sub-task reported by Mayank Bansal and fixed by Vinod Kumar Vavilapalli
  [YARN-321] Merge Patch for YARN-321
  

  Merge Patch

• YARN-1578. Major sub-task reported by Shinichi Yamashita and fixed by Shinichi Yamashita
  Fix how to read history file in FileSystemApplicationHistoryStore
  

  I carried out PiEstimator job at Hadoop cluster which applied YARN-321. After the job end and when I accessed Web UI of HistoryServer, it displayed "500". And HistoryServer daemon log was output as follows. {code} 2014-01-09 13:31:12,227 ERROR org.apache.hadoop.yarn.webapp.Dispatcher: error handling URI: /applicationhistory/appattempt/appattempt_1389146249925_0008_000001 java.lang.reflect.InvocationTargetException at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.yarn.webapp.Dispatcher.service(Dispatcher.java:153) at javax.servlet.http.HttpServlet.service(HttpServlet.java:820) (snip...) Caused by: java.lang.NullPointerException at org.apache.hadoop.yarn.server.applicationhistoryservice.FileSystemApplicationHistoryStore.mergeContainerHistoryData(FileSystemApplicationHistoryStore.java:696) at org.apache.hadoop.yarn.server.applicationhistoryservice.FileSystemApplicationHistoryStore.getContainers(FileSystemApplicationHistoryStore.java:429) at org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryManagerImpl.getContainers(ApplicationHistoryManagerImpl.java:201) at org.apache.hadoop.yarn.server.webapp.AppAttemptBlock.render(AppAttemptBlock.java:110) (snip...) {code} I confirmed that there was container which was not finished from ApplicationHistory file. In ResourceManager daemon log, ResourceManager reserved this container, but did not allocate it. When FileSystemApplicationHistoryStore reads container information without finish data in history file, this problem occurs. In consideration of the case which there is not finish data, we should fix how to read history file in FileSystemApplicationHistoryStore.

• YARN-1577. Blocker sub-task reported by Jian He and fixed by Jian He
  Unmanaged AM is broken because of YARN-1493
  

  Today unmanaged AM client is waiting for app state to be Accepted to launch the AM. This is broken since we changed in YARN-1493 to start the attempt after the application is Accepted. We may need to introduce an attempt state report that client can rely on to query the attempt state and choose to launch the unmanaged AM.

• YARN-1570. Minor improvement reported by Akira AJISAKA and fixed by Akira AJISAKA (documentation)
  Formatting the lines within 80 chars in YarnCommands.apt.vm
  

  In YarnCommands.apt.vm, there are some lines longer than 80 characters. For example: {code} Yarn commands are invoked by the bin/yarn script. Running the yarn script without any arguments prints the description for all commands. {code}

• YARN-1566. Major sub-task reported by Jian He and fixed by Jian He
  Change distributed-shell to retain containers from previous AppAttempt
  

  Change distributed-shell to reuse previous AM's running containers when AM is restarting. It can also be made configurable whether to enable this feature or not.

• YARN-1555. Major sub-task reported by Vinod Kumar Vavilapalli and fixed by Vinod Kumar Vavilapalli
  [YARN-321] Failing tests in org.apache.hadoop.yarn.server.applicationhistoryservice.*
  

  Several tests are failing on the latest YARN-321 branch.

• YARN-1553. Major bug reported by Haohui Mai and fixed by Haohui Mai
  Do not use HttpConfig.isSecure() in YARN
  

  HDFS-5305 and related jira decide that each individual project will have their own configuration on http policy. {{HttpConfig.isSecure}} is a global static method which does not fit the design anymore. The same functionality should be moved into the YARN code base.

• YARN-1536. Minor improvement reported by Karthik Kambatla and fixed by Anubhav Dhoot (resourcemanager)
  Cleanup: Get rid of ResourceManager#get*SecretManager() methods and use the RMContext methods instead
  

  Both ResourceManager and RMContext have methods to access the secret managers, and it should be safe (cleaner) to get rid of the ResourceManager methods.

• YARN-1534. Major sub-task reported by Shinichi Yamashita and fixed by Shinichi Yamashita
  TestAHSWebApp failed in YARN-321 branch
  

  I ran the following commands. And I confirmed failure of TestAHSWebApp. {code} [sinchii@hdX YARN-321-test]$ mvn clean test -Dtest=org.apache.hadoop.yarn.server.applicationhistoryservice.webapp.* {code} {code} Running org.apache.hadoop.yarn.server.applicationhistoryservice.webapp.TestAHSWebServices Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 7.492 sec - in org.apache.hadoop.yarn.server.applicationhistoryservice.webapp.TestAHSWebServices Running org.apache.hadoop.yarn.server.applicationhistoryservice.webapp.TestAHSWebApp Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.193 sec <<< FAILURE! - in org.apache.hadoop.yarn.server.applicationhistoryservice.webapp.TestAHSWebApp initializationError(org.apache.hadoop.yarn.server.applicationhistoryservice.webapp.TestAHSWebApp) Time elapsed: 0.016 sec <<< ERROR! java.lang.Exception: Test class should have exactly one public zero-argument constructor at org.junit.runners.BlockJUnit4ClassRunner.validateZeroArgConstructor(BlockJUnit4ClassRunner.java:144) at org.junit.runners.BlockJUnit4ClassRunner.validateConstructor(BlockJUnit4ClassRunner.java:121) at org.junit.runners.BlockJUnit4ClassRunner.collectInitializationErrors(BlockJUnit4ClassRunner.java:101) at org.junit.runners.ParentRunner.validate(ParentRunner.java:344) (*snip*) {code}

• YARN-1531. Major bug reported by Akira AJISAKA and fixed by Akira AJISAKA (documentation)
  True up yarn command documentation
  

  There are some options which are not written to Yarn Command document. For example, "yarn rmadmin" command options are as follows: {code} Usage: yarn rmadmin -refreshQueues -refreshNodes -refreshSuperUserGroupsConfiguration -refreshUserToGroupsMappings -refreshAdminAcls -refreshServiceAcl -getGroups [username] -help [cmd] -transitionToActive <serviceId> -transitionToStandby <serviceId> -failover [--forcefence] [--forceactive] <serviceId> <serviceId> -getServiceState <serviceId> -checkHealth <serviceId> {code} But some of the new options such as "-getGroups", "-transitionToActive", and "-transitionToStandby" are not documented.

• YARN-1528. Blocker bug reported by Karthik Kambatla and fixed by Karthik Kambatla (resourcemanager)
  Allow setting auth for ZK connections
  

  ZK store and embedded election allow setting ZK-acls but not auth information

• YARN-1525. Major sub-task reported by Xuan Gong and fixed by Cindy Li
  Web UI should redirect to active RM when HA is enabled.
  

  When failover happens, web UI should redirect to the current active rm.

• YARN-1521. Blocker sub-task reported by Xuan Gong and fixed by Xuan Gong
  Mark appropriate protocol methods with the idempotent annotation or AtMostOnce annotation
  

  After YARN-1028, we add the automatically failover into RMProxy. This JIRA is to identify whether we need to add idempotent annotation and which methods can be marked as idempotent.

• YARN-1512. Major improvement reported by Arun C Murthy and fixed by Arun C Murthy
  Enhance CS to decouple scheduling from node heartbeats
  

  Enhance CS to decouple scheduling from node heartbeats; a prototype has improved latency significantly.

• YARN-1493. Major sub-task reported by Jian He and fixed by Jian He
  Schedulers don't recognize apps separately from app-attempts
  

  Today, scheduler is tied to attempt only. We need to separate app-level handling logic in scheduler. We can add new app-level events to the scheduler and separate the app-level logic out. This is good for work-preserving AM restart, RM restart, and also needed for differentiating app-level metrics and attempt-level metrics.

• YARN-1490. Major sub-task reported by Vinod Kumar Vavilapalli and fixed by Jian He
  RM should optionally not kill all containers when an ApplicationMaster exits
  

  This is needed to enable work-preserving AM restart. Some apps can chose to reconnect with old running containers, some may not want to. This should be an option.

• YARN-1470. Major bug reported by Sandy Ryza and fixed by Anubhav Dhoot
  Add audience annotation to MiniYARNCluster
  

  We should make it clear whether this is a public interface.

• YARN-1461. Major sub-task reported by Karthik Kambatla and fixed by Karthik Kambatla (resourcemanager)
  RM API and RM changes to handle tags for running jobs
  

• YARN-1459. Major sub-task reported by Karthik Kambatla and fixed by Xuan Gong (resourcemanager)
  RM services should depend on ConfigurationProvider during startup too
  

  YARN-1667, YARN-1668, YARN-1669 already changed RM to depend on a configuration provider so as to be able to refresh many configuration files across RM fail-over. The dependency on the configuration-provider by the RM should happen at its boot up time too.

• YARN-1452. Major task reported by Zhijie Shen and fixed by Zhijie Shen
  Document the usage of the generic application history and the timeline data service
  

  We need to write a bunch of documents to guide users. such as command line tools, configurations and REST APIs

• YARN-1444. Blocker bug reported by Robert Grandl and fixed by Wangda Tan (client , resourcemanager)
  RM crashes when node resource request sent without corresponding off-switch request
  

  I have tried to force reducers to execute on certain nodes. What I did is I changed for reduce tasks, the RMContainerRequestor#addResourceRequest(req.priority, ResourceRequest.ANY, req.capability) to RMContainerRequestor#addResourceRequest(req.priority, HOST_NAME, req.capability). However, this change lead to RM crashes when reducers needs to be assigned with the following exception: FATAL org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Error in handling event type NODE_UPDATE to the scheduler java.lang.NullPointerException at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainers(LeafQueue.java:841) at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.assignContainersToChildQueues(ParentQueue.java:640) at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.assignContainers(ParentQueue.java:554) at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.nodeUpdate(CapacityScheduler.java:695) at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.handle(CapacityScheduler.java:739) at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.handle(CapacityScheduler.java:86) at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$SchedulerEventDispatcher$EventProcessor.run(ResourceManager.java:549) at java.lang.Thread.run(Thread.java:722)

• YARN-1428. Major sub-task reported by Zhijie Shen and fixed by Zhijie Shen
  RM cannot write the final state of RMApp/RMAppAttempt to the application history store in the transition to the final state
  

  ApplicationFinishData and ApplicationAttemptFinishData are written in the final transitions of RMApp/RMAppAttempt respectively. However, in the transitions, getState() is not getting the state that RMApp/RMAppAttempt is going to enter, but prior one.

• YARN-1417. Blocker bug reported by Omkar Vinit Joshi and fixed by Jian He
  RM may issue expired container tokens to AM while issuing new containers.
  

  Today we create new container token when we create container in RM as a part of schedule cycle. However that container may get reserved or assigned. If the container gets reserved and remains like that (in reserved state) for more than container token expiry interval then RM will end up issuing container with expired token.

• YARN-1410. Major sub-task reported by Bikas Saha and fixed by Xuan Gong
  Handle RM fails over after getApplicationID() and before submitApplication().
  

  App submission involves 1) creating appId 2) using that appId to submit an ApplicationSubmissionContext to the user. The client may have obtained an appId from an RM, the RM may have failed over, and the client may submit the app to the new RM. Since the new RM has a different notion of cluster timestamp (used to create app id) the new RM may reject the app submission resulting in unexpected failure on the client side. The same may happen for other 2 step client API operations.

• YARN-1398. Blocker bug reported by Sunil G and fixed by Vinod Kumar Vavilapalli (resourcemanager)
  Deadlock in capacity scheduler leaf queue and parent queue for getQueueInfo and completedContainer call
  

  getQueueInfo in parentQueue will call child.getQueueInfo(). This will try acquire the leaf queue lock over parent queue lock. Now at same time if a completedContainer call comes and acquired LeafQueue lock and it will wait for ParentQueue's completedConatiner call. This lock usage is not in synchronous and can lead to deadlock. With JCarder, this is showing as a potential deadlock scenario.

• YARN-1389. Major sub-task reported by Mayank Bansal and fixed by Mayank Bansal
  ApplicationClientProtocol and ApplicationHistoryProtocol should expose analogous APIs
  

  As we plan to have the APIs in ApplicationHistoryProtocol to expose the reports of *finished* application attempts and containers, we should do the same for ApplicationClientProtocol, which will return the reports of *running* attempts and containers. Later on, we can improve YarnClient to direct the query of running instance to ApplicationClientProtocol, while that of finished instance to ApplicationHistoryProtocol, making it transparent to the users.

• YARN-1379. Major sub-task reported by Vinod Kumar Vavilapalli and fixed by Vinod Kumar Vavilapalli
  [YARN-321] AHS protocols need to be in yarn proto package name after YARN-1170
  

  Found this while merging YARN-321 to the latest branch-2. Without this, compilation fails.

• YARN-1345. Major sub-task reported by Zhijie Shen and fixed by Zhijie Shen
  Removing FINAL_SAVING from YarnApplicationAttemptState
  

  Whenever YARN-891 is done, we need to add the mapping of RMAppAttemptState.FINAL_SAVING -> YarnApplicationAttemptState.FINAL_SAVING in RMServerUtils#createApplicationAttemptState

• YARN-1301. Minor bug reported by Zhijie Shen and fixed by Tsuyoshi OZAWA
  Need to log the blacklist additions/removals when YarnSchedule#allocate
  

  Now without the log, it's hard to debug whether blacklist is updated on the scheduler side or not

• YARN-1285. Major bug reported by Zhijie Shen and fixed by Kenji Kikushima
  Inconsistency of default "yarn.acl.enable" value
  

  In yarn-default.xml, "yarn.acl.enable" is true while in YarnConfiguration, DEFAULT_YARN_ACL_ENABLE is false.

• YARN-1266. Major sub-task reported by Mayank Bansal and fixed by Mayank Bansal
  Implement PB service and client wrappers for ApplicationHistoryProtocol
  

  Adding ApplicationHistoryProtocolPBService to make web apps to work and changing yarn to run AHS as a seprate process

• YARN-1242. Major sub-task reported by Zhijie Shen and fixed by Mayank Bansal
  Script changes to start AHS as an individual process
  

  Add the command in yarn and yarn.cmd to start and stop AHS

• YARN-1206. Blocker bug reported by Jian He and fixed by Rohith
  AM container log link broken on NM web page even though local container logs are available
  

  With log aggregation disabled, when container is running, its logs link works properly, but after the application is finished, the link shows 'Container does not exist.'

• YARN-1191. Major sub-task reported by Mayank Bansal and fixed by Mayank Bansal
  [YARN-321] Update artifact versions for application history service
  

  Compilation is failing for YARN-321 branch

• YARN-1171. Major improvement reported by Sandy Ryza and fixed by Naren Koneru (documentation , scheduler)
  Add default queue properties to Fair Scheduler documentation
  

  The Fair Scheduler doc is missing the following properties. - defaultMinSharePreemptionTimeout - queueMaxAppsDefault

• YARN-1166. Blocker bug reported by Srimanth Gunturi and fixed by Zhijie Shen (resourcemanager)
  YARN 'appsFailed' metric should be of type 'counter'
  

  Currently in YARN's queue metrics, the cumulative metric 'appsFailed' is of type 'guage' - which means the exact value will be reported. All other cumulative queue metrics (AppsSubmitted, AppsCompleted, AppsKilled) are all of type 'counter' - meaning Ganglia will use slope to provide deltas between time-points. To be consistent, AppsFailed metric should also be of type 'counter'.

• YARN-1123. Major sub-task reported by Zhijie Shen and fixed by Mayank Bansal
  [YARN-321] Adding ContainerReport and Protobuf implementation
  

  Like YARN-978, we need some client-oriented class to expose the container history info. Neither Container nor RMContainer is the right one.

• YARN-1071. Major bug reported by Srimanth Gunturi and fixed by Jian He (resourcemanager)
  ResourceManager's decommissioned and lost node count is 0 after restart
  

  I had 6 nodes in a cluster with 2 NMs stopped. Then I put a host into YARN's {{yarn.resourcemanager.nodes.exclude-path}}. After running {{yarn rmadmin -refreshNodes}}, RM's JMX correctly showed decommissioned node count: {noformat} "NumActiveNMs" : 3, "NumDecommissionedNMs" : 1, "NumLostNMs" : 2, "NumUnhealthyNMs" : 0, "NumRebootedNMs" : 0 {noformat} After restarting RM, the counts were shown as below in JMX. {noformat} "NumActiveNMs" : 3, "NumDecommissionedNMs" : 0, "NumLostNMs" : 0, "NumUnhealthyNMs" : 0, "NumRebootedNMs" : 0 {noformat} Notice that the lost and decommissioned NM counts are both 0.

• YARN-1041. Major sub-task reported by Steve Loughran and fixed by Jian He (resourcemanager)
  Protocol changes for RM to bind and notify a restarted AM of existing containers
  

  For long lived containers we don't want the AM to be a SPOF. When the RM restarts a (failed) AM, it should be given the list of containers it had already been allocated. the AM should then be able to contact the NMs to get details on them. NMs would also need to do any binding of the containers needed to handle a moved/restarted AM.

• YARN-1023. Major sub-task reported by Devaraj K and fixed by Zhijie Shen
  [YARN-321] Webservices REST API's support for Application History
  

• YARN-1017. Blocker sub-task reported by Jian He and fixed by Jian He (resourcemanager)
  Document RM Restart feature
  

  This should give users a general idea about how RM Restart works and how to use RM Restart

• YARN-1007. Major sub-task reported by Devaraj K and fixed by Mayank Bansal
  [YARN-321] Enhance History Reader interface for Containers
  

  If we want to show the containers used by application/app attempt, We need to have two more API's which returns collection of ContainerHistoryData for application id and applcation attempt id something like below. {code:xml} Collection<ContainerHistoryData> getContainers( ApplicationAttemptId appAttemptId); Collection<ContainerHistoryData> getContainers(ApplicationId appId); {code} {code:xml} /** * This method returns {@link Container} for specified {@link ContainerId}. * * @param {@link ContainerId} * @return {@link Container} for ContainerId */ ContainerHistoryData getAMContainer(ContainerId containerId); {code} In the above API, we need to change the argument to application attempt id or we can remove this API because every attempt history data has master container id field, using master container id, history data can get using this below API if it takes argument as container id. {code:xml} /** * This method returns {@link ContainerHistoryData} for specified * {@link ApplicationAttemptId}. * * @param {@link ApplicationAttemptId} * @return {@link ContainerHistoryData} for ApplicationAttemptId */ ContainerHistoryData getContainer(ApplicationAttemptId appAttemptId); {code} Here application attempt can use numbers of containers but we cannot choose which container history data to return. This API argument also need to be changed to take container id instead of app attempt id.

• YARN-987. Major sub-task reported by Mayank Bansal and fixed by Mayank Bansal
  Adding ApplicationHistoryManager responsible for exposing reports to all clients
  

• YARN-986. Blocker sub-task reported by Vinod Kumar Vavilapalli and fixed by Karthik Kambatla
  RM DT token service should have service addresses of both RMs
  

  Previously: YARN should use cluster-id as token service address This needs to be done to support non-ip based fail over of RM. Once the server sets the token service address to be this generic ClusterId/ServiceId, clients can translate it to appropriate final IP and then be able to select tokens via TokenSelectors. Some workarounds for other related issues were put in place at YARN-945.

• YARN-984. Major sub-task reported by Devaraj K and fixed by Devaraj K
  [YARN-321] Move classes from applicationhistoryservice.records.pb.impl package to applicationhistoryservice.records.impl.pb
  

  While creating instance for applicationhistoryservice.records.* pb records, It is throwing the ClassNotFoundException. {code:xml} Caused by: java.lang.ClassNotFoundException: Class org.apache.hadoop.yarn.server.applicationhistoryservice.records.impl.pb.ApplicationHistoryDataPBImpl not found at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1619) at org.apache.hadoop.yarn.factories.impl.pb.RecordFactoryPBImpl.newRecordInstance(RecordFactoryPBImpl.java:56) ... 49 more {code}

• YARN-979. Major sub-task reported by Mayank Bansal and fixed by Mayank Bansal
  [YARN-321] Add more APIs related to ApplicationAttempt and Container in ApplicationHistoryProtocol
  

  ApplicationHistoryProtocol should have the following APIs as well: * getApplicationAttemptReport * getApplicationAttempts * getContainerReport * getContainers The corresponding request and response classes need to be added as well.

• YARN-978. Major sub-task reported by Mayank Bansal and fixed by Mayank Bansal
  [YARN-321] Adding ApplicationAttemptReport and Protobuf implementation
  

  We dont have ApplicationAttemptReport and Protobuf implementation. Adding that. Thanks, Mayank

• YARN-975. Major sub-task reported by Zhijie Shen and fixed by Zhijie Shen
  Add a file-system implementation for history-storage
  

  HDFS implementation should be a standard persistence strategy of history storage

• YARN-974. Major sub-task reported by Zhijie Shen and fixed by Zhijie Shen
  RMContainer should collect more useful information to be recorded in Application-History
  

  To record the history of a container, users may be also interested in the following information: 1. Start Time 2. Stop Time 3. Diagnostic Information 4. URL to the Log File 5. Actually Allocated Resource 6. Actually Assigned Node These should be remembered during the RMContainer's life cycle.

• YARN-967. Major sub-task reported by Devaraj K and fixed by Mayank Bansal
  [YARN-321] Command Line Interface(CLI) for Reading Application History Storage Data
  

• YARN-962. Major sub-task reported by Zhijie Shen and fixed by Zhijie Shen
  Update application_history_service.proto
  

  1. Change it's name to application_history_client.proto 2. Fix the incorrect proto reference. 3. Correct the dir in pom.xml

• YARN-956. Major sub-task reported by Vinod Kumar Vavilapalli and fixed by Zhijie Shen
  [YARN-321] Add a testable in-memory HistoryStorage
  

• YARN-955. Major sub-task reported by Vinod Kumar Vavilapalli and fixed by Mayank Bansal
  [YARN-321] Implementation of ApplicationHistoryProtocol
  

• YARN-954. Major sub-task reported by Vinod Kumar Vavilapalli and fixed by Zhijie Shen
  [YARN-321] History Service should create the webUI and wire it to HistoryStorage
  

• YARN-953. Major sub-task reported by Vinod Kumar Vavilapalli and fixed by Zhijie Shen
  [YARN-321] Enable ResourceManager to write history data
  

• YARN-947. Major sub-task reported by Zhijie Shen and fixed by Zhijie Shen
  Defining the history data classes for the implementation of the reading/writing interface
  

  We need to define the history data classes have the exact fields to be stored. Therefore, all the implementations don't need to have the duplicate logic to exact the required information from RMApp, RMAppAttempt and RMContainer. We use protobuf to define these classes, such that they can be ser/des to/from bytes, which are easier for persistence.

• YARN-935. Major sub-task reported by Zhijie Shen and fixed by Zhijie Shen
  YARN-321 branch is broken due to applicationhistoryserver module's pom.xml
  

  The branch was created from branch-2, hadoop-yarn-server-applicationhistoryserver/pom.xml should use 2.2.0-SNAPSHOT, not 3.0.0-SNAPSHOT. Otherwise, the sub-project cannot be built correctly because of wrong dependency.

• YARN-934. Major sub-task reported by Zhijie Shen and fixed by Zhijie Shen
  HistoryStorage writer interface for Application History Server
  

• YARN-930. Major sub-task reported by Vinod Kumar Vavilapalli and fixed by Vinod Kumar Vavilapalli
  Bootstrap ApplicationHistoryService module
  

• YARN-713. Critical bug reported by Jason Lowe and fixed by Jian He (resourcemanager)
  ResourceManager can exit unexpectedly if DNS is unavailable
  

  As discussed in MAPREDUCE-5261, there's a possibility that a DNS outage could lead to an unhandled exception in the ResourceManager's AsyncDispatcher, and that ultimately would cause the RM to exit. The RM should not exit during DNS hiccups.

• MAPREDUCE-5813. Blocker bug reported by Gera Shegalov and fixed by Gera Shegalov (mrv2 , task)
  YarnChild does not load job.xml with mapreduce.job.classloader=true
  

• MAPREDUCE-5810. Major bug reported by Mit Desai and fixed by Akira AJISAKA (contrib/streaming)
  TestStreamingTaskLog#testStreamingTaskLogWithHadoopCmd is failing
  

• MAPREDUCE-5806. Major bug reported by Eugene Koifman and fixed by Varun Vasudev
  Log4j settings in container-log4j.properties cannot be overridden
  

• MAPREDUCE-5805. Major bug reported by Fengdong Yu and fixed by Akira AJISAKA (jobhistoryserver)
  Unable to parse launch time from job history file
  

• MAPREDUCE-5795. Major bug reported by Yesha Vora and fixed by Xuan Gong
  Job should be marked as Failed if it is recovered from commit.
  

• MAPREDUCE-5794. Minor bug reported by Tsz Wo Nicholas Sze and fixed by Tsz Wo Nicholas Sze (test)
  SliveMapper always uses default FileSystem.
  

• MAPREDUCE-5791. Major bug reported by Nikola Vujic and fixed by Nikola Vujic (client)
  Shuffle phase is slow in Windows - FadviseFileRegion::transferTo does not read disks efficiently
  

• MAPREDUCE-5789. Major bug reported by Rushabh S Shah and fixed by Rushabh S Shah (jobhistoryserver , webapps)
  Average Reduce time is incorrect on Job Overview page
  

• MAPREDUCE-5787. Critical sub-task reported by Rajesh Balamohan and fixed by Rajesh Balamohan (nodemanager)
  Modify ShuffleHandler to support Keep-Alive
  

• MAPREDUCE-5780. Minor bug reported by Tsz Wo Nicholas Sze and fixed by Tsz Wo Nicholas Sze (test)
  SliveTest always uses default FileSystem
  

• MAPREDUCE-5778. Major bug reported by Jason Lowe and fixed by Akira AJISAKA (jobhistoryserver)
  JobSummary does not escape newlines in the job name
  

• MAPREDUCE-5773. Blocker improvement reported by Gera Shegalov and fixed by Gera Shegalov (mr-am)
  Provide dedicated MRAppMaster syslog length limit
  

• MAPREDUCE-5770. Major bug reported by Yesha Vora and fixed by Jian He
  Redirection from AM-URL is broken with HTTPS_ONLY policy
  

• MAPREDUCE-5769. Major bug reported by Rohith and fixed by Rohith
  Unregistration to RM should not be called if AM is crashed before registering with RM
  

• MAPREDUCE-5768. Major bug reported by Zhijie Shen and fixed by Gera Shegalov
  TestMRJobs.testContainerRollingLog fails on trunk
  

• MAPREDUCE-5766. Minor bug reported by Ramya Sunil and fixed by Jian He (applicationmaster)
  Ping messages from attempts should be moved to DEBUG
  

• MAPREDUCE-5761. Trivial improvement reported by Yesha Vora and fixed by Jian He
  Add a log message like "encrypted shuffle is ON" in nodemanager logs
  

• MAPREDUCE-5757. Major bug reported by Jason Lowe and fixed by Jason Lowe (client)
  ConcurrentModificationException in JobControl.toList
  

• MAPREDUCE-5754. Major improvement reported by Gera Shegalov and fixed by Gera Shegalov (jobhistoryserver , mr-am)
  Preserve Job diagnostics in history
  

• MAPREDUCE-5751. Major bug reported by Sangjin Lee and fixed by Sangjin Lee
  MR app master fails to start in some cases if mapreduce.job.classloader is true
  

• MAPREDUCE-5746. Major bug reported by Jason Lowe and fixed by Jason Lowe (jobhistoryserver)
  Job diagnostics can implicate wrong task for a failed job
  

• MAPREDUCE-5732. Major improvement reported by Sandy Ryza and fixed by Sandy Ryza
  Report proper queue when job has been automatically placed
  

• MAPREDUCE-5699. Major bug reported by Karthik Kambatla and fixed by Karthik Kambatla (applicationmaster)
  Allow setting tags on MR jobs
  

• MAPREDUCE-5688. Major bug reported by Mit Desai and fixed by Mit Desai
  TestStagingCleanup fails intermittently with JDK7
  

• MAPREDUCE-5670. Minor bug reported by Jason Lowe and fixed by Chen He (mrv2)
  CombineFileRecordReader should report progress when moving to the next file
  

• MAPREDUCE-5570. Major bug reported by Jason Lowe and fixed by Rushabh S Shah (mr-am , mrv2)
  Map task attempt with fetch failure has incorrect attempt finish time
  

• MAPREDUCE-5553. Minor improvement reported by Paul Han and fixed by Paul Han (applicationmaster)
  Add task state filters on Application/MRJob page for MR Application master
  

• MAPREDUCE-5028. Critical bug reported by Karthik Kambatla and fixed by Karthik Kambatla
  Maps fail when io.sort.mb is set to high value
  

• MAPREDUCE-4052. Major bug reported by xieguiming and fixed by Jian He (job submission)
  Windows eclipse cannot submit job from Windows client to Linux/Unix Hadoop cluster.
  

• MAPREDUCE-2349. Major improvement reported by Joydeep Sen Sarma and fixed by Siddharth Seth (task)
  speed up list[located]status calls from input formats
  

• HDFS-6166. Blocker bug reported by Nathan Roberts and fixed by Nathan Roberts (balancer)
  revisit balancer so_timeout
  

• HDFS-6163. Minor bug reported by Fengdong Yu and fixed by Fengdong Yu (documentation)
  Fix a minor bug in the HA upgrade document
  

• HDFS-6157. Major bug reported by Haohui Mai and fixed by Haohui Mai
  Fix the entry point of OfflineImageViewer for hdfs.cmd
  

• HDFS-6150. Minor improvement reported by Suresh Srinivas and fixed by Suresh Srinivas (namenode)
  Add inode id information in the logs to make debugging easier
  

• HDFS-6140. Major bug reported by Chris Nauroth and fixed by Chris Nauroth (webhdfs)
  WebHDFS cannot create a file with spaces in the name after HA failover changes.
  

• HDFS-6138. Minor improvement reported by Sanjay Radia and fixed by Sanjay Radia (documentation)
  User Guide for how to use viewfs with federation
  

• HDFS-6135. Blocker bug reported by Jing Zhao and fixed by Jing Zhao (journal-node)
  In HDFS upgrade with HA setup, JournalNode cannot handle layout version bump when rolling back
  

• HDFS-6131. Major bug reported by Jing Zhao and fixed by Jing Zhao (documentation)
  Move HDFSHighAvailabilityWithNFS.apt.vm and HDFSHighAvailabilityWithQJM.apt.vm from Yarn to HDFS
  

• HDFS-6130. Blocker bug reported by Fengdong Yu and fixed by Haohui Mai (namenode)
  NPE when upgrading namenode from fsimages older than -32
  

• HDFS-6129. Minor bug reported by Tsz Wo Nicholas Sze and fixed by Tsz Wo Nicholas Sze (datanode)
  When a replica is not found for deletion, do not throw exception.
  

• HDFS-6127. Major bug reported by Arpit Gupta and fixed by Haohui Mai (ha)
  WebHDFS tokens cannot be renewed in HA setup
  

• HDFS-6124. Major sub-task reported by Suresh Srinivas and fixed by Suresh Srinivas
  Add final modifier to class members
  

• HDFS-6123. Minor improvement reported by Tsz Wo Nicholas Sze and fixed by Tsz Wo Nicholas Sze (datanode)
  Improve datanode error messages
  

• HDFS-6120. Major improvement reported by Arpit Agarwal and fixed by Arpit Agarwal (namenode)
  Fix and improve safe mode log messages
  

• HDFS-6117. Minor bug reported by Suresh Srinivas and fixed by Suresh Srinivas (namenode)
  Print file path information in FileNotFoundException
  

• HDFS-6115. Minor bug reported by Vinayakumar B and fixed by Vinayakumar B (datanode)
  flush() should be called for every append on block scan verification log
  

• HDFS-6107. Major bug reported by Colin Patrick McCabe and fixed by Colin Patrick McCabe (datanode)
  When a block can't be cached due to limited space on the DataNode, that block becomes uncacheable
  

• HDFS-6106. Major bug reported by Colin Patrick McCabe and fixed by Colin Patrick McCabe
  Reduce default for dfs.namenode.path.based.cache.refresh.interval.ms
  

• HDFS-6105. Major bug reported by Kihwal Lee and fixed by Haohui Mai
  NN web UI for DN list loads the same jmx page multiple times.
  

• HDFS-6102. Blocker bug reported by Andrew Wang and fixed by Andrew Wang (namenode)
  Lower the default maximum items per directory to fix PB fsimage loading
  

• HDFS-6100. Major bug reported by Arpit Gupta and fixed by Haohui Mai (ha)
  DataNodeWebHdfsMethods does not failover in HA mode
  

• HDFS-6099. Major bug reported by Chris Nauroth and fixed by Chris Nauroth (namenode)
  HDFS file system limits not enforced on renames.
  

• HDFS-6097. Major bug reported by Colin Patrick McCabe and fixed by Colin Patrick McCabe (hdfs-client)
  zero-copy reads are incorrectly disabled on file offsets above 2GB
  

• HDFS-6096. Minor bug reported by Tsz Wo Nicholas Sze and fixed by Tsz Wo Nicholas Sze (test)
  TestWebHdfsTokens may timeout
  

• HDFS-6094. Major bug reported by Arpit Agarwal and fixed by Arpit Agarwal (namenode)
  The same block can be counted twice towards safe mode threshold
  

• HDFS-6090. Minor improvement reported by Akira AJISAKA and fixed by Akira AJISAKA (test)
  Use MiniDFSCluster.Builder instead of deprecated constructors
  

• HDFS-6089. Major bug reported by Arpit Gupta and fixed by Jing Zhao (ha)
  Standby NN while transitioning to active throws a connection refused error when the prior active NN process is suspended
  

• HDFS-6086. Major sub-task reported by Colin Patrick McCabe and fixed by Colin Patrick McCabe (datanode)
  Fix a case where zero-copy or no-checksum reads were not allowed even when the block was cached
  

• HDFS-6085. Major improvement reported by Colin Patrick McCabe and fixed by Colin Patrick McCabe (namenode)
  Improve CacheReplicationMonitor log messages a bit
  

• HDFS-6084. Minor improvement reported by Travis Thompson and fixed by Travis Thompson (namenode)
  Namenode UI - "Hadoop" logo link shouldn't go to hadoop homepage
  

• HDFS-6080. Major improvement reported by Abin Shahab and fixed by Abin Shahab (nfs , performance)
  Improve NFS gateway performance by making rtmax and wtmax configurable
  

• HDFS-6079. Major bug reported by Andrew Wang and fixed by Andrew Wang (hdfs-client)
  Timeout for getFileBlockStorageLocations does not work
  

• HDFS-6078. Minor bug reported by Arpit Agarwal and fixed by Arpit Agarwal (test)
  TestIncrementalBlockReports is flaky
  

• HDFS-6077. Major bug reported by Arpit Gupta and fixed by Jing Zhao
  running slive with webhdfs on secure HA cluster fails with unkown host exception
  

• HDFS-6076. Minor sub-task reported by Tsz Wo Nicholas Sze and fixed by Tsz Wo Nicholas Sze (datanode , test)
  SimulatedDataSet should not create DatanodeRegistration with namenode layout version and type
  

• HDFS-6072. Major improvement reported by Haohui Mai and fixed by Haohui Mai
  Clean up dead code of FSImage
  

• HDFS-6071. Major bug reported by Colin Patrick McCabe and fixed by Colin Patrick McCabe
  BlockReaderLocal doesn't return -1 on EOF when doing a zero-length read on a short file
  

• HDFS-6070. Trivial improvement reported by Andrew Wang and fixed by Andrew Wang
  Cleanup use of ReadStatistics in DFSInputStream
  

• HDFS-6069. Trivial improvement reported by Andrew Wang and fixed by Chris Nauroth (namenode)
  Quash stack traces when ACLs are disabled
  

• HDFS-6068. Major bug reported by Andrew Wang and fixed by sathish (snapshots)
  Disallow snapshot names that are also invalid directory names
  

• HDFS-6067. Major bug reported by Colin Patrick McCabe and fixed by Colin Patrick McCabe (hdfs-client)
  TestPread.testMaxOutHedgedReadPool is flaky
  

• HDFS-6065. Major bug reported by Colin Patrick McCabe and fixed by Colin Patrick McCabe (hdfs-client)
  HDFS zero-copy reads should return null on EOF when doing ZCR
  

• HDFS-6064. Minor bug reported by Vinayakumar B and fixed by Vinayakumar B (datanode)
  DFSConfigKeys.DFS_BLOCKREPORT_INTERVAL_MSEC_DEFAULT is not updated with latest block report interval of 6 hrs
  

• HDFS-6063. Minor bug reported by Colin Patrick McCabe and fixed by Chris Nauroth (test , tools)
  TestAclCLI fails intermittently when running test 24: copyFromLocal
  

• HDFS-6062. Minor bug reported by Jing Zhao and fixed by Jing Zhao
  TestRetryCacheWithHA#testConcat is flaky
  

• HDFS-6061. Major sub-task reported by Colin Patrick McCabe and fixed by Colin Patrick McCabe (datanode)
  Allow dfs.datanode.shared.file.descriptor.path to contain multiple entries and fall back when needed
  

• HDFS-6060. Major sub-task reported by Brandon Li and fixed by Brandon Li (namenode)
  NameNode should not check DataNode layout version
  

• HDFS-6059. Major bug reported by Akira AJISAKA and fixed by Akira AJISAKA
  TestBlockReaderLocal fails if native library is not available
  

• HDFS-6058. Major bug reported by Vinayakumar B and fixed by Haohui Mai
  Fix TestHDFSCLI failures after HADOOP-8691 change
  

• HDFS-6057. Blocker bug reported by Eric Sirianni and fixed by Colin Patrick McCabe (hdfs-client)
  DomainSocketWatcher.watcherThread should be marked as daemon thread
  

• HDFS-6055. Major improvement reported by Suresh Srinivas and fixed by Chris Nauroth (namenode)
  Change default configuration to limit file name length in HDFS
  

  The default configuration of HDFS now sets dfs.namenode.fs-limits.max-component-length to 255 for improved interoperability with other file system implementations. This limits each component of a file system path to a maximum of 255 bytes in UTF-8 encoding. Attempts to create new files that violate this rule will fail with an error. Existing files that violate the rule are not effected. Previously, dfs.namenode.fs-limits.max-component-length was set to 0 (ignored). If necessary, it is possible to set the value back to 0 in the cluster's configuration to restore the old behavior.

• HDFS-6053. Major bug reported by Jing Zhao and fixed by Jing Zhao (namenode)
  Fix TestDecommissioningStatus and TestDecommission in branch-2
  

• HDFS-6051. Blocker bug reported by Chris Nauroth and fixed by Colin Patrick McCabe (hdfs-client)
  HDFS cannot run on Windows since short-circuit shared memory segment changes.
  

• HDFS-6047. Major bug reported by stack and fixed by stack
  TestPread NPE inside in DFSInputStream hedgedFetchBlockByteRange
  

• HDFS-6046. Major sub-task reported by Colin Patrick McCabe and fixed by Colin Patrick McCabe (hdfs-client)
  add dfs.client.mmap.enabled
  

• HDFS-6044. Minor improvement reported by Brandon Li and fixed by Brandon Li (nfs)
  Add property for setting the NFS look up time for users
  

• HDFS-6043. Major improvement reported by Brandon Li and fixed by Brandon Li (nfs)
  Give HDFS daemons NFS3 and Portmap their own OPTS
  

• HDFS-6040. Blocker sub-task reported by Colin Patrick McCabe and fixed by Colin Patrick McCabe (hdfs-client)
  fix DFSClient issue without libhadoop.so and some other ShortCircuitShm cleanups
  

• HDFS-6039. Major bug reported by Yesha Vora and fixed by Chris Nauroth (namenode)
  Uploading a File under a Dir with default acls throws "Duplicated ACLFeature"
  

• HDFS-6038. Major sub-task reported by Haohui Mai and fixed by Jing Zhao (journal-node , namenode)
  Allow JournalNode to handle editlog produced by new release with future layoutversion
  

• HDFS-6033. Major bug reported by Aaron T. Myers and fixed by Aaron T. Myers (caching)
  PBImageXmlWriter incorrectly handles processing cache directives
  

• HDFS-6030. Trivial task reported by Yongjun Zhang and fixed by Yongjun Zhang
  Remove an unused constructor in INode.java
  

• HDFS-6028. Trivial bug reported by Chris Nauroth and fixed by Chris Nauroth (namenode)
  Print clearer error message when user attempts to delete required mask entry from ACL.
  

• HDFS-6025. Minor task reported by Tsz Wo Nicholas Sze and fixed by Tsz Wo Nicholas Sze (build)
  Update findbugsExcludeFile.xml
  

• HDFS-6018. Trivial improvement reported by Jing Zhao and fixed by Jing Zhao
  Exception recorded in LOG when IPCLoggerChannel#close is called
  

• HDFS-6008. Minor bug reported by Benoy Antony and fixed by Benoy Antony (namenode)
  Namenode dead node link is giving HTTP error 500
  

• HDFS-6006. Trivial improvement reported by Akira AJISAKA and fixed by Akira AJISAKA (namenode)
  Remove duplicate code in FSNameSystem#getFileInfo
  

• HDFS-5988. Blocker bug reported by Andrew Wang and fixed by Andrew Wang (namenode)
  Bad fsimage always generated after upgrade
  

• HDFS-5986. Major improvement reported by Suresh Srinivas and fixed by Chris Nauroth (namenode)
  Capture the number of blocks pending deletion on namenode webUI
  

• HDFS-5982. Critical bug reported by Tassapol Athiapinya and fixed by Jing Zhao (namenode)
  Need to update snapshot manager when applying editlog for deleting a snapshottable directory
  

• HDFS-5981. Minor bug reported by Haohui Mai and fixed by Haohui Mai (tools)
  PBImageXmlWriter generates malformed XML
  

• HDFS-5979. Minor improvement reported by Andrew Wang and fixed by Andrew Wang
  Typo and logger fix for fsimage PB code
  

• HDFS-5973. Major sub-task reported by Colin Patrick McCabe and fixed by Colin Patrick McCabe (hdfs-client)
  add DomainSocket#shutdown method
  

• HDFS-5962. Critical bug reported by Kihwal Lee and fixed by Akira AJISAKA
  Mtime and atime are not persisted for symbolic links
  

• HDFS-5961. Critical bug reported by Kihwal Lee and fixed by Kihwal Lee
  OIV cannot load fsimages containing a symbolic link
  

• HDFS-5959. Minor bug reported by Akira AJISAKA and fixed by Akira AJISAKA
  Fix typo at section name in FSImageFormatProtobuf.java
  

• HDFS-5956. Major sub-task reported by Akira AJISAKA and fixed by Akira AJISAKA (tools)
  A file size is multiplied by the replication factor in 'hdfs oiv -p FileDistribution' option
  

• HDFS-5953. Major test reported by Ted Yu and fixed by Akira AJISAKA
  TestBlockReaderFactory fails if libhadoop.so has not been built
  

• HDFS-5950. Major sub-task reported by Colin Patrick McCabe and fixed by Colin Patrick McCabe (datanode , hdfs-client)
  The DFSClient and DataNode should use shared memory segments to communicate short-circuit information
  

• HDFS-5949. Minor bug reported by Travis Thompson and fixed by Travis Thompson (namenode)
  New Namenode UI when trying to download a file, the browser doesn't know the file name
  

• HDFS-5948. Major bug reported by Andrew Wang and fixed by Haohui Mai
  TestBackupNode flakes with port in use error
  

• HDFS-5944. Major bug reported by zhaoyunjiong and fixed by zhaoyunjiong (namenode)
  LeaseManager:findLeaseWithPrefixPath can't handle path like /a/b/ right and cause SecondaryNameNode failed do checkpoint
  

• HDFS-5943. Major bug reported by Yesha Vora and fixed by Suresh Srinivas
  'dfs.namenode.https-address.ns1' property is not used in federation setup
  

• HDFS-5942. Minor sub-task reported by Akira AJISAKA and fixed by Akira AJISAKA (documentation , tools)
  Fix javadoc in OfflineImageViewer
  

• HDFS-5941. Major bug reported by Haohui Mai and fixed by Haohui Mai (documentation , namenode)
  add dfs.namenode.secondary.https-address and dfs.namenode.secondary.https-address in hdfs-default.xml
  

• HDFS-5940. Major sub-task reported by Colin Patrick McCabe and fixed by Colin Patrick McCabe (hdfs-client)
  Minor cleanups to ShortCircuitReplica, FsDatasetCache, and DomainSocketWatcher
  

• HDFS-5939. Major improvement reported by Yongjun Zhang and fixed by Yongjun Zhang (hdfs-client)
  WebHdfs returns misleading error code and logs nothing if trying to create a file with no DNs in cluster
  

• HDFS-5938. Trivial sub-task reported by Colin Patrick McCabe and fixed by Colin Patrick McCabe (hdfs-client)
  Make BlockReaderFactory#BlockReaderPeer a static class
  

• HDFS-5936. Major test reported by Andrew Wang and fixed by Binglin Chang (namenode , test)
  MiniDFSCluster does not clean data left behind by SecondaryNameNode.
  

• HDFS-5935. Minor improvement reported by Travis Thompson and fixed by Travis Thompson (namenode)
  New Namenode UI FS browser should throw smarter error messages
  

• HDFS-5934. Minor bug reported by Travis Thompson and fixed by Travis Thompson (namenode)
  New Namenode UI back button doesn't work as expected
  

• HDFS-5929. Major improvement reported by Siqi Li and fixed by Siqi Li (federation)
  Add Block pool % usage to HDFS federated nn page
  

• HDFS-5922. Major bug reported by Aaron T. Myers and fixed by Arpit Agarwal (datanode)
  DN heartbeat thread can get stuck in tight loop
  

• HDFS-5915. Major bug reported by Haohui Mai and fixed by Haohui Mai (namenode)
  Refactor FSImageFormatProtobuf to simplify cross section reads
  

• HDFS-5913. Minor bug reported by Ted Yu and fixed by Brandon Li (nfs)
  Nfs3Utils#getWccAttr() should check attr parameter against null
  

• HDFS-5910. Major improvement reported by Benoy Antony and fixed by Benoy Antony (security)
  Enhance DataTransferProtocol to allow per-connection choice of encryption/plain-text
  

• HDFS-5904. Major bug reported by Mit Desai and fixed by Mit Desai
  TestFileStatus fails intermittently on trunk and branch2
  

• HDFS-5901. Major bug reported by Vinayakumar B and fixed by Vinayakumar B (namenode)
  NameNode new UI doesn't support IE8 and IE9 on windows 7
  

• HDFS-5900. Major bug reported by Tassapol Athiapinya and fixed by Andrew Wang (caching)
  Cannot set cache pool limit of "unlimited" via CacheAdmin
  

• HDFS-5898. Major sub-task reported by Jing Zhao and fixed by Abin Shahab (nfs)
  Allow NFS gateway to login/relogin from its kerberos keytab
  

• HDFS-5895. Major bug reported by Tassapol Athiapinya and fixed by Tassapol Athiapinya (tools)
  HDFS cacheadmin -listPools has exit_code of 1 when the command returns 0 result.
  

• HDFS-5893. Major bug reported by Yesha Vora and fixed by Haohui Mai
  HftpFileSystem.RangeHeaderUrlOpener uses the default URLConnectionFactory which does not import SSL certificates
  

• HDFS-5892. Minor test reported by Ted Yu and fixed by
  TestDeleteBlockPool fails in branch-2
  

• HDFS-5891. Major bug reported by Haohui Mai and fixed by Haohui Mai (namenode , webhdfs)
  webhdfs should not try connecting the DN during redirection
  

• HDFS-5886. Major bug reported by Ted Yu and fixed by Brandon Li (nfs)
  Potential null pointer deference in RpcProgramNfs3#readlink()
  

• HDFS-5882. Minor test reported by Jimmy Xiang and fixed by Jimmy Xiang
  TestAuditLogs is flaky
  

• HDFS-5881. Critical bug reported by Kihwal Lee and fixed by Kihwal Lee
  Fix skip() of the short-circuit local reader (legacy).
  

• HDFS-5879. Major bug reported by Gera Shegalov and fixed by Gera Shegalov (test)
  Some TestHftpFileSystem tests do not close streams
  

• HDFS-5868. Major sub-task reported by Taylor, Buddy and fixed by (datanode)
  Make hsync implementation pluggable
  

• HDFS-5866. Major sub-task reported by Akira AJISAKA and fixed by Akira AJISAKA (tools)
  '-maxSize' and '-step' option fail in OfflineImageViewer
  

• HDFS-5859. Major bug reported by Colin Patrick McCabe and fixed by Colin Patrick McCabe (datanode)
  DataNode#checkBlockToken should check block tokens even if security is not enabled
  

• HDFS-5857. Major bug reported by Mit Desai and fixed by Mit Desai
  TestWebHDFS#testNamenodeRestart fails intermittently with NPE
  

• HDFS-5856. Minor bug reported by Josh Elser and fixed by Josh Elser (datanode)
  DataNode.checkDiskError might throw NPE
  

• HDFS-5847. Major sub-task reported by Haohui Mai and fixed by Jing Zhao
  Consolidate INodeReference into a separate section
  

• HDFS-5846. Major bug reported by Nikola Vujic and fixed by Nikola Vujic (namenode)
  Assigning DEFAULT_RACK in resolveNetworkLocation method can break data resiliency
  

• HDFS-5843. Major bug reported by Laurent Goujon and fixed by Laurent Goujon (datanode)
  DFSClient.getFileChecksum() throws IOException if checksum is disabled
  

• HDFS-5840. Blocker bug reported by Aaron T. Myers and fixed by Jing Zhao (ha , journal-node , namenode)
  Follow-up to HDFS-5138 to improve error handling during partial upgrade failures
  

• HDFS-5828. Major bug reported by Taylor, Buddy and fixed by Taylor, Buddy (namenode)
  BlockPlacementPolicyWithNodeGroup can place multiple replicas on the same node group when dfs.namenode.avoid.write.stale.datanode is true
  

• HDFS-5821. Major bug reported by Gera Shegalov and fixed by Gera Shegalov (test)
  TestHDFSCLI fails for user names with the dash character
  

• HDFS-5810. Major sub-task reported by Colin Patrick McCabe and fixed by Colin Patrick McCabe (hdfs-client)
  Unify mmap cache and short-circuit file descriptor cache
  

• HDFS-5807. Major bug reported by Mit Desai and fixed by Chen He (test)
  TestBalancerWithNodeGroup.testBalancerWithNodeGroup fails intermittently on Branch-2
  

• HDFS-5804. Major sub-task reported by Abin Shahab and fixed by Abin Shahab (nfs)
  HDFS NFS Gateway fails to mount and proxy when using Kerberos
  

  Fixes NFS on Kerberized cluster.

• HDFS-5803. Major bug reported by Mit Desai and fixed by Chen He
  TestBalancer.testBalancer0 fails
  

• HDFS-5791. Major bug reported by Brandon Li and fixed by Haohui Mai (test)
  TestHttpsFileSystem should use a random port to avoid binding error during testing
  

• HDFS-5790. Major bug reported by Todd Lipcon and fixed by Todd Lipcon (namenode , performance)
  LeaseManager.findPath is very slow when many leases need recovery
  

  Committed to branch-2 and trunk.

• HDFS-5781. Minor improvement reported by Jing Zhao and fixed by Jing Zhao (namenode)
  Use an array to record the mapping between FSEditLogOpCode and the corresponding byte value
  

• HDFS-5780. Major bug reported by Mit Desai and fixed by Mit Desai
  TestRBWBlockInvalidation times out intemittently on branch-2
  

• HDFS-5776. Major improvement reported by Liang Xie and fixed by Liang Xie (hdfs-client)
  Support 'hedged' reads in DFSClient
  

  If a read from a block is slow, start up another parallel, 'hedged' read against a different block replica. We then take the result of which ever read returns first (the outstanding read is cancelled). This 'hedged' read feature will help rein in the outliers, the odd read that takes a long time because it hit a bad patch on the disc, etc. This feature is off by default. To enable this feature, set <code>dfs.client.hedged.read.threadpool.size</code> to a positive number. The threadpool size is how many threads to dedicate to the running of these 'hedged', concurrent reads in your client. Then set <code>dfs.client.hedged.read.threshold.millis</code> to the number of milliseconds to wait before starting up a 'hedged' read. For example, if you set this property to 10, then if a read has not returned within 10 milliseconds, we will start up a new read against a different block replica. This feature emits new metrics: + hedgedReadOps + hedgeReadOpsWin -- how many times the hedged read 'beat' the original read + hedgedReadOpsInCurThread -- how many times we went to do a hedged read but we had to run it in the current thread because dfs.client.hedged.read.threadpool.size was at a maximum.

• HDFS-5775. Major improvement reported by Haohui Mai and fixed by Haohui Mai (namenode)
  Consolidate the code for serialization in CacheManager
  

• HDFS-5768. Major improvement reported by Haohui Mai and fixed by Haohui Mai (namenode)
  Consolidate the serialization code in DelegationTokenSecretManager
  

• HDFS-5767. Blocker bug reported by Yongjun Zhang and fixed by Yongjun Zhang (nfs)
  NFS implementation assumes userName userId mapping to be unique, which is not true sometimes
  

• HDFS-5759. Major bug reported by Haohui Mai and fixed by Haohui Mai
  Web UI does not show up during the period of loading FSImage
  

• HDFS-5746. Major sub-task reported by Colin Patrick McCabe and fixed by Colin Patrick McCabe (datanode , hdfs-client)
  add ShortCircuitSharedMemorySegment
  

• HDFS-5742. Minor bug reported by Arpit Agarwal and fixed by Arpit Agarwal (test)
  DatanodeCluster (mini cluster of DNs) fails to start
  

• HDFS-5726. Minor sub-task reported by Jing Zhao and fixed by Jing Zhao (namenode)
  Fix compilation error in AbstractINodeDiff for JDK7
  

• HDFS-5716. Major bug reported by Haohui Mai and fixed by Haohui Mai (webhdfs)
  Allow WebHDFS to use pluggable authentication filter
  

• HDFS-5715. Major sub-task reported by Jing Zhao and fixed by Jing Zhao (namenode)
  Use Snapshot ID to indicate the corresponding Snapshot for a FileDiff/DirectoryDiff
  

• HDFS-5709. Major improvement reported by Andrew Wang and fixed by Andrew Wang (namenode)
  Improve NameNode upgrade with existing reserved paths and path components
  

• HDFS-5705. Major bug reported by Ted Yu and fixed by Ted Yu (datanode)
  TestSecondaryNameNodeUpgrade#testChangeNsIDFails may fail due to ConcurrentModificationException
  

• HDFS-5698. Major improvement reported by Haohui Mai and fixed by Haohui Mai (namenode)
  Use protobuf to serialize / deserialize FSImage
  

  Use protobuf to serialize/deserialize the FSImage.

• HDFS-5672. Major test reported by Ted Yu and fixed by Jing Zhao (namenode)
  TestHASafeMode#testSafeBlockTracking fails in trunk
  

• HDFS-5647. Major sub-task reported by Haohui Mai and fixed by Haohui Mai (namenode)
  Merge INodeDirectory.Feature and INodeFile.Feature
  

• HDFS-5638. Major sub-task reported by Chris Nauroth and fixed by Vinayakumar B (hdfs-client)
  HDFS implementation of FileContext API for ACLs.
  

• HDFS-5632. Major sub-task reported by Jing Zhao and fixed by Jing Zhao (namenode)
  Add Snapshot feature to INodeDirectory
  

• HDFS-5626. Major bug reported by Stephen Chu and fixed by Colin Patrick McCabe (caching)
  dfsadmin -report shows incorrect cache values
  

• HDFS-5554. Major sub-task reported by Jing Zhao and fixed by Jing Zhao (namenode)
  Add Snapshot Feature to INodeFile
  

• HDFS-5537. Major sub-task reported by Jing Zhao and fixed by Jing Zhao (namenode , snapshots)
  Remove FileWithSnapshot interface
  

• HDFS-5535. Major new feature reported by Nathan Roberts and fixed by Tsz Wo Nicholas Sze (datanode , ha , hdfs-client , namenode)
  Umbrella jira for improved HDFS rolling upgrades
  

• HDFS-5531. Minor sub-task reported by Tsz Wo Nicholas Sze and fixed by Tsz Wo Nicholas Sze (namenode)
  Combine the getNsQuota() and getDsQuota() methods in INode
  

• HDFS-5516. Major bug reported by Chris Nauroth and fixed by Miodrag Radulovic (webhdfs)
  WebHDFS does not require user name when anonymous http requests are disallowed.
  

• HDFS-5492. Minor bug reported by Akira AJISAKA and fixed by Akira AJISAKA (documentation)
  Port HDFS-2069 (Incorrect default trash interval in the docs) to trunk
  

• HDFS-5483. Major sub-task reported by Arpit Agarwal and fixed by Arpit Agarwal (namenode)
  NN should gracefully handle multiple block replicas on same DN
  

• HDFS-5339. Major bug reported by Stephen Chu and fixed by Haohui Mai (webhdfs)
  WebHDFS URI does not accept logical nameservices when security is enabled
  

• HDFS-5321. Major sub-task reported by Haohui Mai and fixed by Haohui Mai
  Clean up the HTTP-related configuration in HDFS
  

  dfs.http.port and dfs.https.port are removed. Filesystem clients, such as WebHdfsFileSystem, now have fixed instead of configurable default ports (i.e., 50070 for http and 50470 for https). Users can explicitly specify the port in the URI to access the file system which runs on non-default ports.

• HDFS-5318. Major improvement reported by Eric Sirianni and fixed by (namenode)
  Support read-only and read-write paths to shared replicas
  

• HDFS-5286. Major sub-task reported by Tsz Wo Nicholas Sze and fixed by Tsz Wo Nicholas Sze (namenode)
  Flatten INodeDirectory hierarchy: add DirectoryWithQuotaFeature
  

• HDFS-5285. Major sub-task reported by Tsz Wo Nicholas Sze and fixed by Jing Zhao (namenode)
  Flatten INodeFile hierarchy: Add UnderContruction Feature
  

• HDFS-5244. Major bug reported by Jinghui Wang and fixed by Jinghui Wang (test)
  TestNNStorageRetentionManager#testPurgeMultipleDirs fails
  

• HDFS-5167. Minor sub-task reported by Jing Zhao and fixed by Tsuyoshi OZAWA (ha , namenode)
  Add metrics about the NameNode retry cache
  

• HDFS-5153. Major improvement reported by Arpit Agarwal and fixed by Arpit Agarwal (datanode)
  Datanode should send block reports for each storage in a separate message
  

• HDFS-5138. Blocker bug reported by Kihwal Lee and fixed by Aaron T. Myers
  Support HDFS upgrade in HA
  

• HDFS-5064. Major bug reported by Aaron T. Myers and fixed by Aaron T. Myers (ha , namenode)
  Standby checkpoints should not block concurrent readers
  

• HDFS-4911. Minor improvement reported by Colin Patrick McCabe and fixed by Colin Patrick McCabe
  Reduce PeerCache timeout to be commensurate with dfs.datanode.socket.reuse.keepalive
  

• HDFS-4858. Minor bug reported by Jagane Sundar and fixed by Henry Wang (datanode)
  HDFS DataNode to NameNode RPC should timeout
  

• HDFS-4685. Major new feature reported by Sachin Jose and fixed by Chris Nauroth (hdfs-client , namenode , security)
  Implementation of ACLs in HDFS
  

  HDFS now supports ACLs (Access Control Lists). ACLs can specify fine-grained file permissions for specific named users or named groups.

• HDFS-4564. Blocker sub-task reported by Daryn Sharp and fixed by Daryn Sharp (webhdfs)
  Webhdfs returns incorrect http response codes for denied operations
  

• HDFS-4370. Major improvement reported by Konstantin Shvachko and fixed by Chu Tong (datanode)
  Fix typo Blanacer in DataNode
  

  I just committed this. Thank you Chu.

• HDFS-4200. Major improvement reported by Suresh Srinivas and fixed by Andrew Wang (datanode)
  Reduce the size of synchronized sections in PacketResponder
  

• HDFS-3969. Major bug reported by Todd Lipcon and fixed by Todd Lipcon (hdfs-client)
  Small bug fixes and improvements for disk locations API
  

• HDFS-3405. Major improvement reported by Aaron T. Myers and fixed by Vinayakumar B
  Checkpointing should use HTTP POST or PUT instead of GET-GET to send merged fsimages
  

• HDFS-3128. Minor bug reported by Eli Collins and fixed by Andrew Wang (test)
  Unit tests should not use a test root in /tmp
  

• HADOOP-10450. Major bug reported by Chris Nauroth and fixed by Chris Nauroth (io , native)
  Build zlib native code bindings in hadoop.dll for Windows.
  

• HADOOP-10449. Minor sub-task reported by Tsz Wo Nicholas Sze and fixed by Tsz Wo Nicholas Sze (security)
  Fix the javac warnings in the security packages.
  

• HADOOP-10442. Blocker bug reported by Kihwal Lee and fixed by Kihwal Lee
  Group look-up can cause segmentation fault when certain JNI-based mapping module is used.
  

• HADOOP-10441. Blocker bug reported by Jing Zhao and fixed by Jing Zhao (metrics)
  Namenode metric "rpc.RetryCache/NameNodeRetryCache.CacheHit" can't be correctly processed by Ganglia
  

• HADOOP-10440. Major bug reported by guodongdong and fixed by guodongdong (fs)
  HarFsInputStream of HarFileSystem, when reading data, computing the position has bug
  

• HADOOP-10437. Minor sub-task reported by Tsz Wo Nicholas Sze and fixed by Tsz Wo Nicholas Sze (conf , util)
  Fix the javac warnings in the conf and the util package
  

• HADOOP-10425. Critical bug reported by Brandon Li and fixed by Tsz Wo Nicholas Sze (fs)
  Incompatible behavior of LocalFileSystem:getContentSummary
  

• HADOOP-10423. Minor improvement reported by Chris Nauroth and fixed by Chris Nauroth (documentation)
  Clarify compatibility policy document for combination of new client and old server.
  

• HADOOP-10422. Minor bug reported by Chris Nauroth and fixed by Chris Nauroth (ipc)
  Remove redundant logging of RPC retry attempts.
  

• HADOOP-10407. Minor sub-task reported by Tsz Wo Nicholas Sze and fixed by Tsz Wo Nicholas Sze (ipc)
  Fix the javac warnings in the ipc package.
  

• HADOOP-10399. Major sub-task reported by Chris Nauroth and fixed by Vinayakumar B (fs)
  FileContext API for ACLs.
  

• HADOOP-10395. Minor bug reported by Arpit Agarwal and fixed by Arpit Agarwal (test)
  TestCallQueueManager is flaky
  

• HADOOP-10394. Major bug reported by Arpit Agarwal and fixed by Arpit Agarwal (test)
  TestAuthenticationFilter is flaky
  

• HADOOP-10393. Minor sub-task reported by Tsz Wo Nicholas Sze and fixed by Tsz Wo Nicholas Sze (security)
  Fix hadoop-auth javac warnings
  

• HADOOP-10386. Minor improvement reported by Arpit Gupta and fixed by Haohui Mai (ha)
  Log proxy hostname in various exceptions being thrown in a HA setup
  

• HADOOP-10383. Major improvement reported by Enis Soztutar and fixed by Enis Soztutar
  InterfaceStability annotations should have RetentionPolicy.RUNTIME
  

• HADOOP-10379. Major improvement reported by Haohui Mai and fixed by Haohui Mai
  Protect authentication cookies with the HttpOnly and Secure flags
  

• HADOOP-10374. Major improvement reported by Enis Soztutar and fixed by Enis Soztutar
  InterfaceAudience annotations should have RetentionPolicy.RUNTIME
  

• HADOOP-10368. Minor bug reported by Ted Yu and fixed by Tsuyoshi OZAWA (util)
  InputStream is not closed in VersionInfo ctor
  

• HADOOP-10355. Major bug reported by Akira AJISAKA and fixed by Haohui Mai
  TestLoadGenerator#testLoadGenerator fails
  

• HADOOP-10353. Major bug reported by Tudor Scurtu and fixed by Tudor Scurtu (fs)
  FsUrlStreamHandlerFactory is not thread safe
  

• HADOOP-10348. Major improvement reported by Haohui Mai and fixed by Haohui Mai
  Deprecate hadoop.ssl.configuration in branch-2, and remove it in trunk
  

• HADOOP-10346. Blocker bug reported by Jason Lowe and fixed by Jason Lowe (security)
  Deadlock while logging tokens
  

• HADOOP-10343. Minor improvement reported by Arpit Gupta and fixed by Arpit Gupta
  Change info to debug log in LossyRetryInvocationHandler
  

• HADOOP-10338. Major bug reported by Andrew Wang and fixed by Colin Patrick McCabe
  Cannot get the FileStatus of the root inode from the new Globber
  

• HADOOP-10337. Major bug reported by Liang Xie and fixed by Liang Xie (metrics)
  ConcurrentModificationException from MetricsDynamicMBeanBase.createMBeanInfo()
  

• HADOOP-10333. Trivial improvement reported by René Nyffenegger and fixed by René Nyffenegger
  Fix grammatical error in overview.html document
  

• HADOOP-10330. Major bug reported by Arpit Agarwal and fixed by Arpit Agarwal (test)
  TestFrameDecoder fails if it cannot bind port 12345
  

• HADOOP-10328. Major bug reported by Arpit Gupta and fixed by Haohui Mai (tools)
  loadGenerator exit code is not reliable
  

• HADOOP-10327. Blocker bug reported by Vinayakumar B and fixed by Vinayakumar B (native)
  Trunk windows build broken after HDFS-5746
  

• HADOOP-10326. Major bug reported by Manuel DE FERRAN and fixed by bc Wong (security)
  M/R jobs can not access S3 if Kerberos is enabled
  

• HADOOP-10320. Trivial bug reported by René Nyffenegger and fixed by René Nyffenegger (documentation)
  Javadoc in InterfaceStability.java lacks final </ul>
  

• HADOOP-10314. Major bug reported by Kihwal Lee and fixed by Rushabh S Shah
  The ls command help still shows outdated 0.16 format.
  

• HADOOP-10301. Blocker bug reported by Daryn Sharp and fixed by Daryn Sharp (security)
  AuthenticationFilter should return Forbidden for failed authentication
  

• HADOOP-10295. Major improvement reported by Jing Zhao and fixed by Jing Zhao (tools/distcp)
  Allow distcp to automatically identify the checksum type of source files and use it for the target
  

  Add option for distcp to preserve the checksum type of the source files. Users can use "-pc" as distcp command option to preserve the checksum type.

• HADOOP-10285. Major sub-task reported by Chris Li and fixed by
  Admin interface to swap callqueue at runtime
  

• HADOOP-10280. Major sub-task reported by Chris Li and fixed by Chris Li
  Make Schedulables return a configurable identity of user or group
  

• HADOOP-10278. Major sub-task reported by Chris Li and fixed by Chris Li (ipc)
  Refactor to make CallQueue pluggable
  

• HADOOP-10249. Major bug reported by Dilli Arumugam and fixed by Dilli Arumugam
  LdapGroupsMapping should trim ldap password read from file
  

• HADOOP-10221. Major improvement reported by Benoy Antony and fixed by Benoy Antony (security)
  Add a plugin to specify SaslProperties for RPC protocol based on connection properties
  

• HADOOP-10211. Major improvement reported by Benoy Antony and fixed by Benoy Antony (security)
  Enable RPC protocol to negotiate SASL-QOP values between clients and servers
  

  The hadoop.rpc.protection configuration property previously supported specifying a single value: one of authentication, integrity or privacy. An unrecognized value was silently assumed to mean authentication. This configuration property now accepts a comma-separated list of any of the 3 values, and unrecognized values are rejected with an error. Existing configurations containing an invalid value must be corrected. If the property is empty or not specified, authentication is assumed.

• HADOOP-10191. Blocker bug reported by Gera Shegalov and fixed by Gera Shegalov (viewfs)
  Missing executable permission on viewfs internal dirs
  

• HADOOP-10184. Major new feature reported by Chris Nauroth and fixed by Chris Nauroth (fs , security)
  Hadoop Common changes required to support HDFS ACLs.
  

• HADOOP-10139. Major improvement reported by Akira AJISAKA and fixed by Akira AJISAKA (documentation)
  Update and improve the Single Cluster Setup document
  

• HADOOP-10085. Blocker bug reported by Karthik Kambatla and fixed by Steve Loughran
  CompositeService should allow adding services while being inited
  

• HADOOP-10070. Major bug reported by Aaron T. Myers and fixed by Aaron T. Myers (security)
  RPC client doesn't use per-connection conf to determine server's expected Kerberos principal name
  

• HADOOP-10015. Minor bug reported by Haohui Mai and fixed by Nicolas Liochon (security)
  UserGroupInformation prints out excessive ERROR warnings
  

• HADOOP-9525. Major test reported by Ivan Mitic and fixed by Ivan Mitic (test , util)
  Add tests that validate winutils chmod behavior on folders
  

• HADOOP-9454. Major improvement reported by Jordan Mendelson and fixed by Akira AJISAKA (fs/s3)
  Support multipart uploads for s3native
  

• HADOOP-8691. Minor improvement reported by Jason Lowe and fixed by Daryn Sharp (fs)
  FsShell can print "Found xxx items" unnecessarily often
  

  The `ls` command only prints "Found foo items" once when listing the directories recursively.