dolphinscheduler部署文档

参考文档：https://blog.csdn.net/weixin_47884711/article/details/123523144

一、环境

平台：HDP(Ambari)

zookeeper：192.168.3.135,192.168.3.169,192.168.3.221

yarn：192.168.3.221,192.168.3.169

mysql：192.168.0.140

dolphinscheduler版本：1.3.5

mysql驱动：mysql-connector-java-8.0.22.jar

配置目录：/data/soft/dolphinscheduler-incubating-1.3.5

分发节点信息：192.168.3.203

二、集群搭建

1、在203节点新建安装目录

mkdir -p /data/soft/

2、将tar包导入安装目录并解压

[root@node-str-coretrrl0003 soft]# pwd

/data/soft

apache-dolphinscheduler-incubating-1.3.5-dolphinscheduler-bin.tar.gz

tar -zxvf apapache-dolphinscheduler-incubating-1.3.5-dolphinscheduler-bin.tar.gz

3、解压包重命名

mv apache-dolphinscheduler-1.3.6-bin dolphinscheduler-1.3.6-bin

[root@node-str-coretrrl0003 soft]# ls

mv apache-dolphinscheduler-incubating-1.3.5-dolphinscheduler-bin dolphinscheduler-incubating-1.3.5

4、每个节点配置hosts映射

[root@node-str-coretrrl0003 soft]# cat /etc/hosts

::1 localhost localhost.localdomain localhost6 localhost6.localdomain6

127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4

5、配置ssh免密登录 192.168.0.203分发密钥到下面机器

192.168.3.203

6、数据库初始化,华为云rds 192.168.0.140

（1）将mysql驱动放入DS的lib目录下；

[root@node-str-coretrrl0003 lib]# ll mysql-connector-java-5.1.47.jar

-rwxr-x—. 1 root root 1007502 Jul 22 11:00 mysql-connector-java-5.1.47.jar

[root@node-str-coretrrl0003 lib]# pwd

/data/soft/dolphinscheduler-incubating-1.3.5/lib

（2）建库授权；

mysql> CREATE DATABASE dolphinscheduler DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;

mysql> GRANT ALL PRIVILEGES ON dolphinscheduler.* TO ‘dolph’@‘%’ IDENTIFIED BY ‘***********’;

mysql> GRANT ALL PRIVILEGES ON dolphinscheduler.* TO ‘dolph’@‘localhost’ IDENTIFIED BY ‘{password}’;

mysql> flush privileges;

7、配置conf/datasource.properties

mysql

spring.datasource.driver-class-name=com.mysql.jdbc.Driver

spring.datasource.url=jdbc:mysql://192.168.0.140:3306/dolphinscheduler?characterEncoding=UTF-8&allowMultiQueries=true

spring.datasource.username=dolph

spring.datasource.password=**********

修改并保存完后，执行 script 目录下的创建表及导入基础数据脚本：

sh script/create-dolphinscheduler.sh

8、修改conf/env/dolphinscheduler_env.sh

export HADOOP_HOME=/opt/client/HDFS/hadoop

export HADOOP_CONF_DIR=/opt/client/HDFS/hadoop/etc/hadoop

export SPARK_HOME=/opt/client/Spark/spark

export PYTHON_HOME=/opt/soft/python

export JAVA_HOME=/opt/client/JDK/jdk

export HIVE_HOME=/opt/client/Hive/Beeline

export FLINK_HOME=/root/flink-1.11.2-hdp-2.8.3

export PATH=$HADOO{P}_{H}OME/bin:$SPARK_HOME1/bin:$SPAR{K}_{H}OME2/bin:$PYTHON_HOME:$JAV{A}_{H}OME/bin:$HIVE_HOME/bin:$PATH:$FLINK_HOME/bin:$DATA{X}_{H}OME:$PATH

用不到的可以暂且注释；

flink-1.11.2-hdp-2.8.3.tar rz上传到/root

tar -zxvf /root/flink-1.11.2-hdp-2.8.3.tar

9、将 JDK 软链到 /usr/bin/java 下

#sudo ln -s /usr/java/jdk1.8.0_212-amd64/bin/java /usr/bin/java

sudo ln -s /opt/client/JDK/jdk /opt/Bigdata/jdk1.8.0_232

10、修改conf/config/install_config.conf 中的各参数

[root@node-str-coretrrl0003 config]# grep -Ev “^$|#” install_config.conf

dbtype=“mysql”

dbhost=“192.168.0.140:3306”

username=“dolph”

dbname=“dolphinscheduler”

password=“*********”

zkQuorum=“192.168.3.135:2181,192.168.3.169:2181,192.168.3.221:2181”

installPath=“/opt/Bigdata/dolphinscheduler-1.3.5”

deployUser=“root”

mailServerHost=“smtp.exmail.qq.com”

mailServerPort=“25”

mailSender=“xxxxxxxxxx”

mailUser=“xxxxxxxxxx”

mailPassword=“xxxxxxxxxx”

starttlsEnable=“true”

sslEnable=“false”

sslTrust=“smtp.exmail.qq.com”

resourceStorageType=“HDFS”

defaultFS=“hdfs://hacluster”

s3Endpoint=“http://192.168.xx.xx:9010”

s3AccessKey=“xxxxxxxxxx”

s3SecretKey=“xxxxxxxxxx”

yarnHaIps=“192.168.3.221,192.168.3.169”

resourceUploadPath=“/dolphinscheduler”

hdfsRootUser=“root”

kerberosStartUp=“false”

krb5ConfPath=“$installPath/conf/krb5.conf”

keytabUserName=“hdfs-mycluster@ESZ.COM”

keytabPath=“$installPath/conf/hdfs.headless.keytab”

apiServerPort=“12345”

ips=“192.168.3.50,192.168.3.32,192.168.3.222”

sshPort=“22”

masters=“192.168.3.135,192.168.3.83”

workers=“192.168.3.50:default,192.168.3.32:default,192.168.3.222:default”

alertServer=“192.168.3.50”

apiServers=“192.168.3.50”

11.修改conf/common.properties配置

[root@node-str-coretrrl0003 conf]# grep -Ev “^$|#” common.properties
resource.storage.type=HDFS
resource.upload.path=/dolphinscheduler
hadoop.security.authentication.startup.state=false
java.security.krb5.conf.path=/opt/Bigdata/dolphinscheduler-1.3.5/conf/krb5.conf
login.user.keytab.username=hdfs-mycluster@ESZ.COM
login.user.keytab.path=/opt/Bigdata/dolphinscheduler-1.3.5/conf/hdfs.headless.keytab
hdfs.root.user=root
fs.defaultFS=hdfs://hacluster
fs.s3a.endpoint=http://192.168.xx.xx:9010
fs.s3a.access.key=xxxxxxxxxx
fs.s3a.secret.key=xxxxxxxxxx
yarn.resourcemanager.ha.rm.ids=192.168.3.221,192.168.3.169
yarn.application.status.address=http://:8088/ws/v1/cluster/apps/%s
development.state=false
kerberos.expire.time=7

12 执行install.sh

/data/soft/dolphinscheduler-incubating-1.3.5/install.sh

13. 访问地址：http://192.168.3.50:12345/dolphinscheduler/ui/view/login/index.html

admin