MaWo: A Master Worker framework on YARN Services

Overview

MaWo is a YARN service based framework which handles Master Worker based workload. This is an app which can take an input job specification with tasks, their expected durations and have a Master dish the tasks off to a predetermined set of workers. The components will be responsible to finish the job within specific time duration.

MaWo Components

MaWo app is a YARN Service Application. It has mainly two components.

  • Master
  • Read MaWo-Payload file and create a queue of Tasks
  • Register Worker
  • Assign tasks to worker nodes
  • Monitor status of Tasks
  • Log Task status

  • Worker

  • Send heartbeat to Worker
  • Execute Task