Package org.apache.hadoop.mapreduce.lib.output.committer.manifest

Class ManifestCommitterStatisticNames

java.lang.Object
org.apache.hadoop.mapreduce.lib.output.committer.manifest.ManifestCommitterStatisticNames
@Public @Unstable public final class ManifestCommitterStatisticNames extends Object
Statistic names for committers. Please keep in sync with org.apache.hadoop.fs.s3a.Statistic so that S3A and manifest committers are in sync.

  • Field Details

    • COMMITTER_BYTES_COMMITTED_COUNT

      public static final String COMMITTER_BYTES_COMMITTED_COUNT
      Amount of data committed: "committer_bytes_committed".
      See Also:

    • COMMITTER_COMMIT_JOB

      public static final String COMMITTER_COMMIT_JOB
      Duration Tracking of time to commit an entire job: "committer_commit_job".
      See Also:

    • COMMITTER_FILES_COMMITTED_COUNT

      public static final String COMMITTER_FILES_COMMITTED_COUNT
      Number of files committed: "committer_files_committed".
      See Also:

    • COMMITTER_TASKS_COMPLETED_COUNT

      public static final String COMMITTER_TASKS_COMPLETED_COUNT
      "Count of successful tasks:: "committer_tasks_completed".
      See Also:

    • COMMITTER_TASKS_FAILED_COUNT

      public static final String COMMITTER_TASKS_FAILED_COUNT
      Count of failed tasks: "committer_tasks_failed".
      See Also:

    • COMMITTER_COMMITS_ABORTED_COUNT

      public static final String COMMITTER_COMMITS_ABORTED_COUNT
      Count of commits aborted: "committer_commits_aborted".
      See Also:

    • COMMITTER_COMMITS_REVERTED_COUNT

      public static final String COMMITTER_COMMITS_REVERTED_COUNT
      Count of commits reverted: "committer_commits_reverted".
      See Also:

    • COMMITTER_COMMITS_FAILED

      public static final String COMMITTER_COMMITS_FAILED
      Count of commits failed: "committer_commits.failures".
      See Also:

    • COMMITTER_FILE_COUNT_MEAN

      public static final String COMMITTER_FILE_COUNT_MEAN
      The number of files in a task. This will be a MeanStatistic.
      See Also:

    • COMMITTER_FILE_SIZE_MEAN

      public static final String COMMITTER_FILE_SIZE_MEAN
      File Size.
      See Also:

    • COMMITTER_TASK_DIRECTORY_COUNT_MEAN

      public static final String COMMITTER_TASK_DIRECTORY_COUNT_MEAN
      What is a task attempt's directory count.
      See Also:

    • COMMITTER_TASK_DIRECTORY_DEPTH_MEAN

      public static final String COMMITTER_TASK_DIRECTORY_DEPTH_MEAN
      What is the depth of a task attempt's directory tree.
      See Also:

    • COMMITTER_TASK_FILE_COUNT_MEAN

      public static final String COMMITTER_TASK_FILE_COUNT_MEAN
      The number of files in a task. This will be a MeanStatistic.
      See Also:

    • COMMITTER_TASK_FILE_SIZE_MEAN

      public static final String COMMITTER_TASK_FILE_SIZE_MEAN
      The number of files in a task. This will be a MeanStatistic.
      See Also:

    • COMMITTER_TASK_MANIFEST_FILE_SIZE

      public static final String COMMITTER_TASK_MANIFEST_FILE_SIZE
      The size of manifest files. This will be a MeanStatistic. Useful to highlight whether this size is too large and effort to optimise for file IO and memory consumption justified.
      See Also:

    • OP_COMMIT_FILE_RENAME

      public static final String OP_COMMIT_FILE_RENAME
      A file renamed during the commit operation "commit_file_rename".
      See Also:

    • OP_COMMIT_FILE_RENAME_RECOVERED

      public static final String OP_COMMIT_FILE_RENAME_RECOVERED
      A file rename during the commit operation was recovered from a failure "commit_file_rename_recovered".
      See Also:

    • OP_CREATE_DIRECTORIES

      public static final String OP_CREATE_DIRECTORIES
      Directory creation "op_create_directories".
      See Also:

    • OP_CREATE_ONE_DIRECTORY

      public static final String OP_CREATE_ONE_DIRECTORY
      Creating a single directory "op_create_one_directory".
      See Also:

    • OP_DELETE_FILE_UNDER_DESTINATION

      public static final String OP_DELETE_FILE_UNDER_DESTINATION
      A file in the destination directory tree has been deleted. "op_delete_file_under_destination".
      See Also:

    • OP_DIRECTORY_SCAN

      public static final String OP_DIRECTORY_SCAN
      Directory scan "op_directory_scan".
      See Also:

    • OP_STAGE_JOB_COMMIT

      public static final String OP_STAGE_JOB_COMMIT
      Overall job commit "committer_commit_job".
      See Also:

    • OP_LOAD_ALL_MANIFESTS

      public static final String OP_LOAD_ALL_MANIFESTS
      "op_load_all_manifests".
      See Also:

    • OP_LOAD_MANIFEST

      public static final String OP_LOAD_MANIFEST
      Load a task manifest: "op_load_manifest".
      See Also:

    • OP_MKDIRS_RETURNED_FALSE

      public static final String OP_MKDIRS_RETURNED_FALSE
      mkdir failure statistic: "op_mkdir_returned_false". This is incremented whenever mkdir() returns false, e.g because there was a file at the end of the path.
      See Also:

    • OP_MSYNC

      public static final String OP_MSYNC
      msync statistic: "op_msync". This should be the same as StoreStatisticNames.OP_MSYNC; it's duplicated just to isolate this committer into a single JAR for ease of testing.
      See Also:

    • OP_PREPARE_DIR_ANCESTORS

      public static final String OP_PREPARE_DIR_ANCESTORS
      Preparing ancestor dirs: "op_prepare_dir_ancestors". Probing for paths being files, and if true: deleting them.
      See Also:

    • OP_RENAME_DIR

      public static final String OP_RENAME_DIR
      Rename a dir: "op_rename".
      See Also:

    • OP_RENAME_FILE

      public static final String OP_RENAME_FILE
      Rename a file: "op_rename".
      See Also:

    • OP_SAVE_TASK_MANIFEST

      public static final String OP_SAVE_TASK_MANIFEST
      Save a task manifest: "task_stage_save_task_manifest".
      See Also:

    • OP_SAVE_SUMMARY_FILE

      public static final String OP_SAVE_SUMMARY_FILE
      Save a summary file: "task_stage_save_summary_file".
      See Also:

    • OP_STAGE_TASK_ABORT_TASK

      public static final String OP_STAGE_TASK_ABORT_TASK
      Task abort: "task_stage_abort_task".
      See Also:

    • OP_STAGE_JOB_ABORT

      public static final String OP_STAGE_JOB_ABORT
      Job abort: "job_stage_abort".
      See Also:

    • OP_STAGE_JOB_CLEANUP

      public static final String OP_STAGE_JOB_CLEANUP
      Job cleanup: "job_stage_cleanup".
      See Also:

    • OP_STAGE_JOB_CREATE_TARGET_DIRS

      public static final String OP_STAGE_JOB_CREATE_TARGET_DIRS
      Prepare Directories Stage: "job_stage_create_target_dirs".
      See Also:

    • OP_STAGE_JOB_LOAD_MANIFESTS

      public static final String OP_STAGE_JOB_LOAD_MANIFESTS
      Load Manifest Stage: "job_stage_load_manifests".
      See Also:

    • OP_STAGE_JOB_RENAME_FILES

      public static final String OP_STAGE_JOB_RENAME_FILES
      Rename files stage duration: "job_stage_rename_files".
      See Also:

    • OP_STAGE_JOB_SETUP

      public static final String OP_STAGE_JOB_SETUP
      Job Setup Stage: "job_stage_setup".
      See Also:

    • OP_STAGE_JOB_SAVE_SUCCESS

      public static final String OP_STAGE_JOB_SAVE_SUCCESS
      Job saving _SUCCESS marker Stage: "job_stage_save_success_marker".
      See Also:

    • OP_STAGE_JOB_VALIDATE_OUTPUT

      public static final String OP_STAGE_JOB_VALIDATE_OUTPUT
      Output Validation (within job commit) Stage: "job_stage_optional_validate_output".
      See Also:

    • OP_STAGE_TASK_SAVE_MANIFEST

      public static final String OP_STAGE_TASK_SAVE_MANIFEST
      Task saving manifest file Stage: "task_stage_save_manifest".
      See Also:

    • OP_STAGE_TASK_SETUP

      public static final String OP_STAGE_TASK_SETUP
      Task Setup Stage: "task_stage_setup".
      See Also:

    • OP_STAGE_TASK_COMMIT

      public static final String OP_STAGE_TASK_COMMIT
      Task Commit Stage: "task_stage_commit".
      See Also:

    • OP_STAGE_TASK_SCAN_DIRECTORY

      public static final String OP_STAGE_TASK_SCAN_DIRECTORY
      Task Scan directory Stage: "task_stage_scan_directory".
      See Also:

    • OP_DELETE_DIR

      public static final String OP_DELETE_DIR
      Delete a directory: "op_delete_dir".
      See Also: