Spark on Yarn With K8s

方案1：

• 1.hadooptool131
  CDH部署hdfs 、yarn、 hbase的gateway⻆⾊(其实目的是为了客户端配置文件)

• 2.keytab⽂件权限(每个节点)

[root@hadooptool131 ~]# chmod 777 /etc/kerberos/*.keytab
1

• 3.Apache Spark配置CDH Hadoop目录

[root@hadooptool131 ~]# vi /opt/software/spark/spark-2.4.0-bin-hadoop2.6/conf/spark-env.sh
HADOOP_CONF_DIR=/etc/hadoop/conf
YARN_CONF_DIR=/etc/hadoop/conf
SPARK_HOME=/opt/software/spark/spark
SPARK_CONF_DIR=${SPARK_HOME}/conf
1
2
3
4
5

• 4.hbase-site.xml做软连接到Hadoop配置目录

[root@hadooptool131 ~]# ln -s /etc/hbase/conf/hbase-site.xml /etc/hadoop/conf/hbase-site.xml
1

• 5.配置hdfs⽬录的/user权限

[root@hadooptool131 ~]# kinit hdfs
Password for hdfs@HADOOP.COM:
[root@hadooptool131 ~]# hdfs dfs -chmod -R 777 /user
1
2
3

• 6.cdh yarn的配置

min.user.id : 0
banned.users : yarn mapred bin
allowed.system.users : nobody impala hive llama hdfs hbase
1
2
3

• 7.重启yarn服务，⽣效配置

• 8.提交jar

spark提交到yarn平台，需要principal、keytab参数
[root@hadooptool131 ~]# klist
klist: No credentials cache found (filename: /tmp/krb5cc_0)
1
2
3

• 自带Pi案例提交

${SPARK_HOME}/bin/spark-submit \
--master yarn \
--deploy-mode cluster \
--queue ruozedata \
--driver-memory 1G \
--num-executors 1 \
--executor-memory 1G \
--executor-cores 1 \
--principal hdfs@HADOOP.COM \
--keytab /etc/kerberos/hdfs.keytab \
--class org.apache.spark.examples.SparkPi \
/root/spark-2.4.2-bin-hadoop2.6/examples/jars/spark-examples_2.12-2.4.2.jar
1
2
3
4
5
6
7
8
9
10
11
12

• 生产ETL标准提交参数

${SPARK_HOME}/spark-submit \
--master yarn \
--deploy-mode cluster \
--queue ruozedata \
--driver-memory 4G \
--num-executors 10 \
--executor-memory 6G \
--executor-cores 3 \
--principal hdfs@HADOOP.COM \    
--keytab /etc/kerberos/hdfs.keytab \  
--class com.ruozedata.homepage.CarrierAmount \
--conf "spark.yarn.archive=hdfs://ruozedata/spark/sparkjar20220324.zip" \
--conf "spark.yarn.am.memory=1024m" \
--conf "spark.yarn.am.memoryOverhead=1024m" \
--conf "spark.driver.memoryOverhead=1024m" \
--conf "spark.executor.memoryOverhead=1024m" \

--conf "spark.streaming.backpressure.enabled=true" \
--conf "spark.streaming.kafka.maxRatePerPartition=1250" \
--conf "spark.locality.wait=10s" \
--conf "spark.executor.heartbeatInterval=360000" \
--conf "spark.network.timeout=420000" \
--conf "spark.serializer=org.apache.spark.serializer.KryoSerializer"
--conf "spark.hadoop.fs.hdfs.impl.disable.cache=true" \

--conf "spark.yarn.am.extraJavaOptions=-XX:+UseG1GC -XX:MaxGCPauseMillis=300 -XX:InitiatingHeapOccupancyPercent=50 -XX:G1ReservePercent=20 -XX:+DisableExplicitGC -Dcdh.version=5.16.1" \
--conf "spark.driver.extraJavaOptions=-XX:+UseG1GC -XX:MaxGCPauseMillis=300 -XX:InitiatingHeapOccupancyPercent=50 -XX:G1ReservePercent=20 -XX:+DisableExplicitGC -Dcdh.version=5.16.1" \
--conf "spark.executor.extraJavaOptions=-XX:+UseG1GC -XX:MaxGCPauseMillis=300 -XX:InitiatingHeapOccupancyPercent=50 -XX:G1ReservePercent=20 -XX:+DisableExplicitGC -Dcdh.version=5.16.1" \

/opt/maintaim/core/ruozedata-1.0.jar
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30

---

方案2：

Spark和Flink均为分布式计算引擎，在使用yarn作为资源调度器提交任务并且连接拥有Kerberos认证的Hbase时，同时面临着认证文件分发与获取的问题。两者的解决方案也是类似的，现在driver端加载认证文件，存储到分布式缓存，然后再Executor端获取文件目录进行认证.

Spark on Yarn
driver端

  sparkSession.sparkContext.addFile(keytabPath)
  sparkSession.sparkContext.addFile(krb5Path)
1
2

keytabPath为keytab文件的在driver的绝对目录  
krb5Path为krb5.conf文件在driver的绝对目录

executor端

val krb5Path=SparkFiles.get(krb5FileName) 
val keytabPath=SparkFiles.get(keytabFileName)
1
2

krb5FileName为krb5.conf文件名   
keytabFileName为keytab文件名
krb5Path和keytabPath则是executor端配置文件的目录