Spark

From Dave's wiki
Jump to navigation Jump to search

Setup

To Setup an Apache Spark Cluster, we need to setup the master and worker nodes.

https://medium.com/codex/setup-a-spark-cluster-step-by-step-in-10-minutes-922c06f8e2b1

sudo apt install openjdk-8-jdk
sudo apt install python3
curl -O https://apache.claz.org/spark/spark-3.1.1/spark-3.1.1-bin-hadoop3.2.tgz
tar xvf spark-3.1.1-bin-hadoop3.2.tgz
sudo mv spark-3.1.1-bin-hadoop3.2/ /opt/spark
export SPARK_HOME=/opt/spark
export PATH=$PATH:$SPARK_HOME/bin:$SPARK_HOME/sbin
cd /opt/spark/conf
SPARK_MASTER_HOST=192.168.0.1
start-master.sh
stop-master.sh
start-worker.sh spark://192.168.0.123:7077