Package org.apache.hadoop.mapred.nativetask

Class Platform

java.lang.Object
org.apache.hadoop.mapred.nativetask.Platform
@Public @Evolving public abstract class Platform extends Object
Base class for platforms. A platform is a framework running on top of MapReduce, like Hadoop, Hive, Pig, Mahout. Each framework defines its own key type and value type across a MapReduce job. For each platform, we should implement serializers such that we could communicate data with native side and native comparators so our native output collectors could sort them and write out. We've already provided the HadoopPlatform that supports all key types of Hadoop and users could implement their custom platform.

  • Field Summary

    Fields
    Modifier and Type
    Field
    Description
    protected Set<String>
    keyClassNames
     

  • Constructor Summary

    Constructors
    Constructor
    Description
    Platform()
     

  • Method Summary

    Modifier and Type
    Method
    Description
    protected abstract boolean
    define(Class<?> keyComparator)
    whether it's the platform that has defined a custom Java comparator NativeTask doesn't support custom Java comparators (set with mapreduce.job.output.key.comparator.class) but a platform (e.g Pig) could also set that conf and implement native comparators so we shouldn't bail out.
    abstract void
    init()
    initialize a platform, where we should call registerKey
    abstract String
    name()
     
    protected void
    registerKey(String keyClassName, Class<?> key)
    associate a key class with its serializer and platform
    protected abstract boolean
    support(String keyClassName, INativeSerializer<?> serializer, JobConf job)
    whether a platform supports a specific key should at least satisfy two conditions 1. the key belongs to the platform 2. the associated serializer must implement INativeComparable interface

    Methods inherited from class java.lang.Object

    clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

  • Field Details

    • keyClassNames

      protected Set<String> keyClassNames

  • Constructor Details

    • Platform

      public Platform()

  • Method Details

    • init

      public abstract void init() throws IOException
      initialize a platform, where we should call registerKey
      Throws:
      IOException

    • name

      public abstract String name()
      Returns:
      name of a Platform, useful for logs and debug

    • registerKey

      protected void registerKey(String keyClassName, Class<?> key) throws IOException
      associate a key class with its serializer and platform
      Parameters:
      keyClassName - map out key class name
      key - key serializer class
      Throws:
      IOException

    • support

      protected abstract boolean support(String keyClassName, INativeSerializer<?> serializer, JobConf job)
      whether a platform supports a specific key should at least satisfy two conditions 1. the key belongs to the platform 2. the associated serializer must implement INativeComparable interface
      Parameters:
      keyClassName - map out put key class name
      serializer - serializer associated with key via registerKey
      job - job configuration
      Returns:
      true if the platform has implemented native comparators of the key and false otherwise

    • define

      protected abstract boolean define(Class<?> keyComparator)
      whether it's the platform that has defined a custom Java comparator NativeTask doesn't support custom Java comparators (set with mapreduce.job.output.key.comparator.class) but a platform (e.g Pig) could also set that conf and implement native comparators so we shouldn't bail out.
      Parameters:
      keyComparator - comparator set with mapreduce.job.output.key.comparator.class