Upcoming Activities


Qcon 2019, Beijing

MAY 06–08, 2019

Apache Hadoop machine learning engine Submarine and ecology.

Submarine is a machine learning platform jointly developed by the Hadoop and Zeppelin communities. It supports Tensorflow, and machine learning frameworks such as Pytorch run in Kubernetes and YARN in a stand-alone or distributed manner.

Now you can use the Submarine-installer to easily install and deploy NVIDIA-Docker, ETCD, Calico and other machines to learn the running environment. In Zeppelin, you can visualize the interactive notebook before the Spark machine learning data processing. Develop and validate Tensorflow’s Pythone algorithm, complete data processing of machine learning jobs and model training in Zeppelin’s Workflow for full-link, and periodically perform offline model training in Kubernetes/Hadoop.

More infomation: https://2019.qconbeijing.com/presentation/1440


DataWorks Summit in Barcelona, Spain

MARCH 18–21, 2019

Hadoop {Submarine} Project: Running Deep Learning Workloads on YARN.

Deep learning is useful for enterprises tasks in the field of speech recognition, image classification, AI chatbots and machine translation, just to name a few.

In order to train deep learning/machine learning models, applications such as TensorFlow / MXNet / Caffe / XGBoost can be leveraged. And sometimes these applications will be used together to solve different problems.

To make distributed deep learning/machine learning applications easily launched, managed, monitored. Hadoop community has introduced Submarine project along with other improvements such as first-class GPU support, container-DNS support, scheduling improvements, etc. These improvements make distributed deep learning/machine learning applications run on YARN as simple as running it locally, which can let machine-learning engineers focus on algorithms instead of worrying about underlying infrastructure. Also, YARN can better manage a shared cluster which runs deep learning/machine learning and other services/ETL jobs with these improvements.

In this session, we will take a closer look at Submarine project as well as other improvements and show how to run these deep learning workloads on YARN with demos. Audiences can start trying running these workloads on YARN after this talk.

More infomation: https://dataworkssummit.com/barcelona-2019/session/hadoop-submarine-project-running-deep-learning-workloads-on-yarn/