bigdata-dev team mailing list archive

[Kevin W Monroe <kevin.monroe@xxxxxxxxxxxxx>]

Kevin W Monroe has proposed merging lp:~bigdata-dev/charms/trusty/apache-spark/trunk into lp:charms/trusty/apache-spark.

Requested reviews:
  Juju Big Data Development (bigdata-dev)

For more details, see:
https://code.launchpad.net/~bigdata-dev/charms/trusty/apache-spark/trunk/+merge/275100

- pull resources from our s3 bucket
- update test script to only run sparkpi in the mode that spark is configured (as set by the MASTER envar)
-- 
Your team Juju Big Data Development is requested to review the proposed merge of lp:~bigdata-dev/charms/trusty/apache-spark/trunk into lp:charms/trusty/apache-spark.

=== modified file 'resources.yaml'
--- resources.yaml	2015-10-08 15:16:59 +0000
+++ resources.yaml	2015-10-20 20:41:43 +0000
@@ -11,6 +11,6 @@
     hash: 204fc101f3c336692feeb1401225407c575a90d42af5d7ab83937b59c42a6af9
     hash_type: sha256
   spark-x86_64:
-    url: http://www.apache.org/dist/spark/spark-1.3.0/spark-1.3.0-bin-hadoop2.4.tgz
+    url: https://s3.amazonaws.com/jujubigdata/apache/x86_64/spark-1.3.0-bin-hadoop2.4-094b5116.tgz
     hash: 094b5116231b6fec8d3991492c06683126ce66a17f910d82780a1a9106b41547
     hash_type: sha256

=== modified file 'scripts/sparkpi.sh'
--- scripts/sparkpi.sh	2015-06-25 17:43:10 +0000
+++ scripts/sparkpi.sh	2015-10-20 20:41:43 +0000
@@ -1,9 +1,8 @@
 #!/bin/bash
 
-echo "Running SparkPi in yarn-cluster mode"
-spark-submit --class org.apache.spark.examples.SparkPi --master yarn-cluster /usr/lib/spark/lib/spark-examples*.jar 10
-echo ""
-
-echo "Running SparkPi in yarn-client mode"
-spark-submit --class org.apache.spark.examples.SparkPi --master yarn-client /usr/lib/spark/lib/spark-examples*.jar 10
+if [[ -z $MASTER ]]; then
+  MASTER='yarn-client'
+fi
+echo "Running SparkPi in ${MASTER} mode"
+spark-submit --class org.apache.spark.examples.SparkPi --master ${MASTER} /usr/lib/spark/lib/spark-examples*.jar 10
 echo ""