Spark Demo 1 - A simple Euler sum computation¶
This is a very trivial demo of spark. If you download and install spark from spark-2.0.2-bin-hadoop2.7 then cd to that directory and do the following
$ export PYSPARK_DRIVER_PYTHON=Jupyter $ export PYSPARK_DRIVER_PYTHON_OPTS=notebook $bin/pyspark
Jupyter will come up in that directory with the spark context "sc" already loaded. The usual way to get tthe spark context is
sc = pyspark.SparkContext('local[*]') which tell spark to use all available cores on the local machine. For other clusters you may need something else.
