Apache Hadoop 3.1.1 Release Notes

These release notes cover new developer and user-facing incompatibilities, important issues, features, and major improvements.

This change updates the Microsoft Windows build directions to be more flexible with regards to Visual Studio compiler versions:

Additionally, Snappy and ISA-L that use bin as the location of the DLL will now be recognized without having to set their respective lib paths if the prefix is set.

Note to contributors:

It is very important that solutions for any patches remain at the VS 2010-level.

WASB: Bug fix to support non-sequential page blob reads. Required for HBASE replication.

WASB: Bug fix for recent regression in hflush() and hsync().

New command is added to dfsadmin. hdfs dfsadmin [-upgrade [query | finalize] 1. -upgrade query gives the upgradeStatus 2. -upgrade finalize is equivalent to -finalizeUpgrade.

WASB: Fix Spark process hang at shutdown due to use of non-daemon threads by updating Azure Storage Java SDK to 7.0

Mover could have fail after 20+ minutes if a block move was enqueued for this long, between two DataNodes due to an internal constant that was introduced for Balancer, but affected Mover as well. The internal constant can be configured with the dfs.balancer.max-iteration-time parameter after the patch, and affects only the Balancer. Default is 20 minutes.

WASB: listStatus 10x performance improvement for listing 700,000 files