Need to update Spark 1.6.0

[quanghoc]

Hi

The Spark version is 1.3.1 in this VM:

/usr/local/bin/spark-1.3.1-bin-hadoop2.6/

I need to do an update to 1.6.0. How is Spark being installed inside the VM and is there instruction to update? Or do you plan to push an update soon?

[jameswinegar]

It is being installed from the vagrantfile. If I remember correctly it pulls down an image which means you'd have to upgrade yourself.

Spark can be run standalone fairly easily depending on what you're trying to accomplish.

On Jan 25, 2016, at 2:27 AM, quanghoc [email protected] wrote:

Hi

The Spark version is 1.3.1 in this VM. I need to do an update to 1.6.0. Where is Spark being installed inside the VM?

—
Reply to this email directly or view it on GitHub.

[eduarger]

I share the same doubt. How can we upgrade the spark version? I would like to use all the features (jupyter notebooks, for example). Because I have dowloaded a pre built version, and it works fine, but the jupyter still use the old spark release, how can we force Jupyter to use the new spark version?

[felixcheung]

Unfortunately this is for the MOOC course in the summer of 2015, and is no longer being updated.
Please search on github for many alternatives: https://github.com/search?utf8=%E2%9C%93&q=vagrant+spark&type=Repositories&ref=searchresults

[sliuproxy]

It can be upgrade simply by replacing the files. First, download the latest version of spark with command:
wget http://apache.mirror.digitalpacific.com.au/spark/spark-1.6.1/spark-1.6.1-bin-hadoop2.6.tgz
then unzip it by
tar -xvf spark-1.6.1-bin-hadoop2.6.tgz
then rename the old spark version by (or you can remove it)
sudo mv /usr/local/bin/spark-1.3.1-bin-hadoop2.6 /usr/local/bin/spark-1.3.1-bin-hadoop2.6.old
then we move the new version to this place (note that the destination folder is still 1.3.1):
sudo mv spark-1.6.1-bin-hadoop2.6 /usr/local/bin/spark-1.3.1-bin-hadoop2.6
change the permissions for spark by
sudo chmod -R 777 /usr/local/bin/spark-1.3.1-bin-hadoop2.6
shutdown the virtual machine (not save state, we need a restart), then check the new version by
spark-submit –version
you should see version 1.6.1 displayed.

[eduarger]

@swliuproxy : thanks very much for your information, it work like charm!

[q15928]

@swliuproxy 👍 Thanks for that.
However I also updated IPython with the following codes
sudo pip install -U ipython
sudo pip install jupyter
After that, I can't open the notebook anymore

[sliuproxy]

@q15928 If you upgraded jupyter to version 4.0, then you can't use spark directly as jupyter 4.0 doesn't support "profile" feature.

A work around will be install findspark package
sudo pip install findspark
then start a normal python notebook, type in the code

import findspark
findspark.init('/path/to/spark_home')
import pyspark
sc = pyspark.SparkContext(appName="myAppName")

Better to use the original old version jupyter for the purpose of this course.