Using the CLI MiniCluster, users can simply start and stop a single-node Hadoop cluster with a single command, and without the need to set any environment variables or manage configuration files. The CLI MiniCluster starts both a YARN/MapReduce & HDFS clusters.
This is useful for cases where users want to quickly experiment with a real Hadoop cluster or test non-Java programs that rely on significant Hadoop functionality.
You should be able to obtain the Hadoop tarball from the release. Also, you can directly create a tarball from the source:
$ mvn clean install -DskipTests $ mvn package -Pdist -Dtar -DskipTests -Dmaven.javadoc.skip
NOTE: You will need protoc 2.5.0 installed.
The tarball should be available in hadoop-dist/target/ directory.
From inside the root directory of the extracted tarball, you can start the CLI MiniCluster using the following command:
$ bin/mapred minicluster -rmport RM_PORT -jhsport JHS_PORT
In the example command above, RM_PORT and JHS_PORT should be replaced by the user’s choice of these port numbers. If not specified, random free ports will be used.
There are a number of command line arguments that the users can use to control which services to start, and to pass other configuration properties. The available command line arguments:
$ -D <property=value> Options to pass into configuration object $ -datanodes <arg> How many datanodes to start (default 1) $ -format Format the DFS (default false) $ -help Prints option help. $ -jhsport <arg> JobHistoryServer port (default 0--we choose) $ -namenode <arg> URL of the namenode (default is either the DFS $ cluster or a temporary dir) $ -nnport <arg> NameNode port (default 0--we choose) $ -nnhttpport <arg> NameNode HTTP port (default 0--we choose) $ -nodemanagers <arg> How many nodemanagers to start (default 1) $ -nodfs Don't start a mini DFS cluster $ -nomr Don't start a mini MR cluster $ -rmport <arg> ResourceManager port (default 0--we choose) $ -writeConfig <path> Save configuration to this XML file. $ -writeDetails <path> Write basic information to this JSON file.
To display this full list of available arguments, the user can pass the -help argument to the above command.