When upgrading to HDP 2.3.4 using Ambari, Spark 1.4.1 is automatically upgraded to 1.5.2. However, if you wish to return to using 1.4.1:
Remove Spark 1.5.2 from your HDP cluster using Ambari:
curl -u admin:admin -H "X-Requested-By: ambari" -X DELETE
http://<AMBARI_HOST>:8080/api/v1/clusters/<CLUSTER_NAME>/services/SPARK
Manually install Spark 1.4.1 with HDP 2.3.0 Installing HDP Manually: Installing and Configuring Apache Spark.
