Chapter 24. Installing Apache Slider

Run the following command to install Slider.

As the root user, edit the following properties in the /etc/hadoop/conf/yarn-site.xml file.

<property>
     <name>hadoop.registry.zk.quorum</name>
     <value>$ZOOKEEPERQUORUM-SERVERS</value>
     <description>List of hostname:port pairs defining the zookeeper quorum binding for the registry
     </description>
</property>
 
<property>
     <name>hadoop.registry.rm.enabled</name>
     <value>true</value>
     <description> Is the registry enabled: does the RM start it up, create the user
        and system paths, and purge service records when containers, application attempts
        and applications complete?
     </description>
</property>

Set hadoop.registry.rm.enabled to true and set hadoop.registry.zk.quorum to the address and port number of your ZooKeeper Quorum server (usually assigned to port 2181). For example:

<property>
     <name>hadoop.registry.zk.quorum</name>
     <value>node-1.example.com:2181</value>
     <description>List of hostname:port pairs defining the zookeeper quorum binding for the registry
     </description>
</property>
 
<property>
     <name>hadoop.registry.rm.enabled</name>
     <value>true</value>
     <description>Is the registry enabled: does the RM start it up, create the user
        and system paths, and purge service records when containers, application attempts
        and applications complete?
     </description>
</property>

As the root user, specify the JAVA_HOME and HADOOP_CONF_DIR settings in the /etc/slider/conf/slider-env.sh file. For example:

# this is the shell script to start Slider deploying an application
# Usage: slider <action> <commands>

# The env variable SLIDER_JVM_OPTS can be used to override
# the default JVM opts

export JAVA_HOME=/usr/hadoop-jdk1.6.0_31
export HADOOP_CONF_DIR=/etc/hadoop/conf

Use the following command to switch to the slider bin directory:

cd /usr/hdp/current/slider-client/bin

Use the Slider version command to verify that Slider has installed properly:

./slider version

Ensure that there are no errors, and that your results say “Compiled against Hadoop <current_hadoop_version>”.

[root@node-1 bin]# ./slider version
2014-10-27 14:42:45,340 [main] INFO client.SliderClient - Slider Core-0.51.0.2.3.4.0 Built against commit# d766e78d77 on Java 1.6.0_31 by jenkins
2014-10-27 14:42:45,351 [main] INFO client.SliderClient - Compiled against Hadoop 2.3.4.0-$BUILD
2014-10-27 14:42:45,375 [main] INFO client.SliderClient - Hadoop runtime version (no branch) with source checksum 517963c273a1f4f8f5bfc15d92aa013 and build date 2014-10-27T03:27Z
2014-10-27 14:42:45,383 [main] INFO util.ExitUtil - Exiting with status 0
[root@node-1 bin]# 

If HDP is installed in a cluster without using Ambari, invoke the following command (as user hdfs) after the slider client is installed:

slider dependency --upload

The above command uploads the Slider dependency tarball in an HDP specific path:

/hdp/apps/<hdp_version>/slider/slider.tar.gz

All subsequent Slider application creation jobs use this dependency tarball. The benefit is twofold. The application submission time reduces drastically from approximately 10-20 seconds to approximately 1-2 seconds, depending on the hardware. It also eliminates all potential jar conflict issues when 3rd party client applications use the slider-core jar to submit Slider application creation requests.