Spark安装

Spark安装

解压spark

tar –zxvf spark-1.6.1-bin-hadoop2.6.tgz

配置环境变量

export SPARK_HOME=/usr/tools/spark-1.6.1-bin-hadoop2.6

export PATH=$PATH:$SPARK_HOME/bin

是环境变量生效

source /etc/profile

Spark配置

进入spark的conf目录

cd /usr/tools/spark-1.6.1-bin-hadoop2.6/conf

cp spark-env.sh.template spark-env.sh

cp log4j.properties.template log4j.properties

cp slaves.template slaves

编辑spark-env.sh

export SCALA_HOME=/usr/tools/scala-2.11.8

export JAVA_HOME=/usr/tools/jdk1.7.0_67

export SPARK_WORKER_MEMORY=1G

export HADOOP_CONF_DIR=/usr/tools/hadoop-2.6.4/etc/hadoop

编辑slaves

jokeros1

jokeros2

jokeros3

将spark拷贝到子节点上然后配置环境变量并使其生效。

scp -r spark-1.6.1-bin-hadoop2.6 root@jokeros2:/usr/tools

scp -r spark-1.6.1-bin-hadoop2.6 root@jokeros3:/usr/tools

进入主节点的sbin目录

运行start-all.sh

主节点上Master Worker两个进程，子节点上Worker一个进程

http://jokeros1:8080

http://192.168.0.142:4040/jobs

export SPARK_DAEMON_JAVA_OPTS="-Dspark.deploy.recoveryMode=ZOOKEEPER -Dspark.deploy.zookeeper.url=hdm1:2181,hdm2:2181,hdm3:2181 -Dspark.deploy.zookeeper.dir=/usr/hadoop/spark-2.0.1-bin-hadoop2.6/zookeeper"