27. Configuring and Upgrading Apache Spark

Instructions in this section are specific to HDP-2.3.4 and later. For earlier versions for HDP, refer to the version-specific documentation.

Add the node where you want Apache Spark 1.5.2 History Server to run. Install the version corresponding to the HDP version you currently have installed.
1. su - root
2. yum install spark_spark_2_3_4_0_3371-master -y
3. To use Python: yum install spark_spark_2_3_4_0_3371-python
4. conf-select create-conf-dir --package spark --stack-version spark_2_3_4_0_3371 --conf-version 0
5. cp /etc/spark/spark_2_3_4_0_3371/0/* /etc/spark/spark_2_3_4_0_3371/0/
6. conf-select set-conf-dir --package spark --stack-version spark_2_3_4_0_3371 --conf-version 0
7. hdp-select set spark-client spark_2_3_4_0_3371
8. hdp-select set spark-historyserver spark_2_3_4_0_3371
Validate the Spark installation. As user spark, run SparkPI example:
1. sudo su spark
2. cd /usr/hdp/current/spark-client
3. ./bin/run-example SparkPi 10
Restart Spark on YARN in either yarn-cluster mode or yarn-client mode:
- yarn-cluster mode: ./usr/hdp/current/spark-client/bin/spark-submit --class path.to.your.Class --master yarn-cluster [options] [app options]
- yarn-client mode: ./usr/hdp/current/spark-client/bin/spark-shell --master yarn-client