DolpinScheduler

1 简介

官网：https://dolphinscheduler.apache.org/zh-cn/

文档：https://dolphinscheduler.apache.org/zh-cn/docs/

https://dolphinscheduler.apache.org/zh-cn/docs/2.0.5/user_doc/guide/flink-call.html

Apache DolphinScheduler是一个分布式 易扩展的可视化工作流任务调度平台，致力于解决数据处理流程中错综复杂的关系，使调度系统在数据处理流程中开箱即用。

版本：2.0.5

1.1 特性

• 简单易用：DAG监控界面，所有流程定义都是可视化，通过拖拽任务完成定制DAG，通过API方式与第三方系统集成，一键部署。
• 高可靠性：去中心化的多Master和多Worker服务对等架构，避免单Master压力过大，另外采用任务缓冲队列来避免过载。
• 高扩展性：支持自定义任务类型，调度器使用分布式调度，调度能力随集群线性增长，Master和Worker支持动态上下线。
• 丰富的使用场景：支持多租户，支持暂停恢复操作，紧密贴合大数据生态，提供Spark、Hive、MapRedece、Python、Sub_process、Shell等近20中任务类型。

1.2 核心架构（主要角色）

• MasterServer：采用分布式务中心设计理念，MasterServer主要负责DAG任务切分，任务提交、任务监控，并同时监听其他MasterServer和WorkerServer的健康状态。（负责工作流）
• WorkerServer：也采用分布式务中心设计理念，WorkerServer主要负责任务的执行和提供日志服务。（负责任务）
• ZooKeeper：系统中MasterServer和WorkerServer节点都通过ZooKeeper来进行集群管理和容错。
• Alert服务：提供告警相关服务。
• API接口层：主要负责处理前端UI层的请求。
• UI：系统的前端页面，提供系统的各种可视化操作界面。

2 部署说明

2.1 软硬件要求

操作系统要求

服务器硬件要求

2.2 部署模式

2.2.1 单机模式

单机模式（sandalone）模式下，所有服务均集中于一个StandaloneServer进程中，并且其中内置了注册中心ZooKeeper和数据库H2。只需配置JDK环境，即可一键启动DolphinScheduler。（学习、测试）

2.2.2 伪集群模式

伪集群模式（Pseudo-Cluster）是单台机器部署DolphinSchdeuler各项服务，该模式下master、worker、api server、logger server等服务都只在同一台机器上。ZooKeeper和数据库需要单独安装并进行响应配置。

2.2.3 集群模式

相对于伪集群，区别在于可以在多台机器部署 DolphinScheduler各项服务，并且Maser、Worker等服务可配置多个。

3 集群搭建

下载：https://dlcdn.apache.org/dolphinscheduler/2.0.5/

3.1 环境准备

3.1.1 硬件

Master：通常2~3个
Worker：多个

这里配置一个Master、三个Worker

3.1.2 软件

1. 三台阶段均需安装JDK（1.8+），并配置相关环境变量。

2. 需部署数据库，支持MySQL（5.7+）或者PostgreSQL（8.2.15+）

3. 需部署Zookeeper（3.4.6+）

4. 三台节点均需安装进程树分析工具psmisc

   sudo yum install -y psmisc
   1

本次环境：
JAVA：V1.8.0_151
ZooKeeper：V3.7.1
驱动：
mysql-connector-java-8.0.16.jar

3.2 创建元数据库及用户

安装docker：https://blog.csdn.net/zx77588023/article/details/122652560

docker run -d \
-p 3306:3306 \
-v /usr/local/mysql/data/conf:/etc/mysql/conf.d \
-v /usr/local/mysql/data/data:/var/lib/mysql \
-e MYSQL_ROOT_PASSWORD=root \
--name ds_mysql mysql:5.7
1
2
3
4
5
6

登录mysql

mysql -u root -p
1

1、创建数据库

CREATE DATABASE dolphinscheduler DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
1

2、创建用户

CREATE USER 'dolphinscheduler'@'%' IDENTIFIED BY 'dolphinscheduler';
1

如果测试使用，提供密码过于简单，可以降低密码强度级别

set global validate_password_length=4;
set global validate_password_policy=0;
1
2

3、赋予用户响应权限

GRANT ALL PRIVILEGES ON dolphinscheduler.* TO 'dolphinscheduler'@'%';
1

flush privileges;
1

3.3 初始化数据库

1、拷贝MySQL驱动到DolphinScheduler的解压目录lib中，要求版本：JDBC Driver 8.0.16

cp mysql-connector-java-8.0.16.jar /usr/local/ds/apache-dolphinscheduler-2.0.5-bin/lib
1

2、执行数据库初始化脚本

cd /usr/local/ds/apache-dolphinscheduler-2.0.5-bin
./script/create-dolphinscheduler.sh
1
2

3.4 配置一键部署脚本

三台机器均配置免密超级用户 charles

useradd charles -u 520
1

sudo vim /etc/sudoers
1

charles ALL=(ALL)       NOPASSWD:ALL
1

# 创建用户需使用 root 登录
useradd dolphinscheduler

# 添加密码
echo "dolphinscheduler" | passwd --stdin dolphinscheduler

# 配置 sudo 免密
sed -i '$adolphinscheduler  ALL=(ALL)  NOPASSWD: NOPASSWD: ALL' /etc/sudoers
sed -i 's/Defaults    requirett/#Defaults    requirett/g' /etc/sudoers

# 修改目录权限，使得部署用户对二进制包解压后的 apache-dolphinscheduler-*-bin 目录有操作权限
chown -R dolphinscheduler:dolphinscheduler apache-dolphinscheduler-*-bin
1
2
3
4
5
6
7
8
9
10
11
12

修改 conf/config目录下得install_config.conf文件

vim conf/config/install_config.conf
1

# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements.  See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License.  You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#

# ---------------------------------------------------------
# INSTALL MACHINE
# ---------------------------------------------------------
# A comma separated list of machine hostname or IP would be installed DolphinScheduler,
# including master, worker, api, alert. If you want to deploy in pseudo-distributed
# mode, just write a pseudo-distributed hostname
# Example for hostnames: ips="ds1,ds2,ds3,ds4,ds5", Example for IPs: ips="192.168.8.1,192.168.8.2,192.168.8.3,192.168.8.4,192.168.8.5"
# 将要部署的任一 DolpinScheduler 服务的主机名或ip列表
ips="master,slave1,slave2"

# Port of SSH protocol, default value is 22. For now we only support same port in all `ips` machine
# modify it if you use different ssh port
sshPort="22"

# A comma separated list of machine hostname or IP would be installed Master server, it
# must be a subset of configuration `ips`.
# Example for hostnames: masters="ds1,ds2", Example for IPs: masters="192.168.8.1,192.168.8.2"
# master所在主机名列表，必须是 ips 的子集
masters="master"

# A comma separated list of machine : or :.All hostname or IP must be a
# subset of configuration `ips`, And workerGroup have default value as `default`, but we recommend you declare behind the hosts
# Example for hostnames: workers="ds1:default,ds2:default,ds3:default", Example for IPs: workers="192.168.8.1:default,192.168.8.2:default,192.168.8.3:default"
# work 主机名及队列，此处的ip 必须在 ips 列表中
workers="master:default,slave1:default,slave2:default"

# A comma separated list of machine hostname or IP would be installed Alert server, it
# must be a subset of configuration `ips`.
# Example for hostname: alertServer="ds3", Example for IP: alertServer="192.168.8.3"
# 告警服务所在服务器主机名
alertServer="slave1"

# A comma separated list of machine hostname or IP would be installed API server, it
# must be a subset of configuration `ips`.
# Example for hostname: apiServers="ds1", Example for IP: apiServers="192.168.8.1"
# api 服务所在服务器主机名
apiServers="slave1"

# A comma separated list of machine hostname or IP would be installed Python gateway server, it
# must be a subset of configuration `ips`.
# Example for hostname: pythonGatewayServers="ds1", Example for IP: pythonGatewayServers="192.168.8.1"

# 不需要的配置项，可以保留默认值，也可以 #注释
#pythonGatewayServers="ds1"

# The directory to install DolphinScheduler for all machine we config above. It will automatically be created by `install.sh` script if not exists.
# Do not set this configuration same as the current path (pwd)
# 安装路径
installPath="/usr/local/ds/ds_install"

# The user to deploy DolphinScheduler for all machine we config above. For now user must create by yourself before running `install.sh`
# script. The user needs to have sudo privileges and permissions to operate hdfs. If hdfs is enabled than the root directory needs
# to be created by this user

# 部署用户 具有sudo权限，配置免密 
deployUser="charles"
#deployUser="root"

# The directory to store local data for all machine we config above. Make sure user `deployUser` have permissions to read and write this directory.

# 临时路径
dataBasedirPath="/tmp/dolphinscheduler"

# ---------------------------------------------------------
# DolphinScheduler ENV
# ---------------------------------------------------------
# JAVA_HOME, we recommend use same JAVA_HOME in all machine you going to install DolphinScheduler
# and this configuration only support one parameter so far.
javaHome="/usr/local/jdk1.8.0_151"

# DolphinScheduler API service port, also this is your DolphinScheduler UI component's URL port, default value is 12345

apiServerPort="12345"

# ---------------------------------------------------------
# Database
# NOTICE: If database value has special characters, such as `.*[]^${}\+?|()@#&`, Please add prefix `\` for escaping.
# ---------------------------------------------------------
# The type for the metadata database
# Supported values: ``postgresql``, ``mysql`, `h2``.

# 数据库类型
DATABASE_TYPE="mysql"

# string jdbc:mysql://127.0.0.1:3306/dolphinscheduler?useUnicode=true&characterEncoding=UTF-8 as example
SPRING_DATASOURCE_URL="jdbc:mysql://master:3306/dolphinscheduler?useUnicode=true&characterEncoding=UTF-8"

# 数据库用户名
SPRING_DATASOURCE_USERNAME="dolphinscheduler"

# 数据库密码
SPRING_DATASOURCE_PASSWORD="dolphinscheduler"

# ---------------------------------------------------------
# Registry Server(这里配置zookeeper)
# ---------------------------------------------------------
# Registry Server plugin name, should be a substring of `registryPluginDir`, DolphinScheduler use this for verifying configuration consistency
registryPluginName="zookeeper"

# Registry Server address.
registryServers="master:2181,slave1:2181,slave2:2181"

# Registry Namespace（注册中心命名空间）
registryNamespace="dolphinscheduler"

# ---------------------------------------------------------
# Worker Task Server
# ---------------------------------------------------------
# Worker Task Server plugin dir. DolphinScheduler will find and load the worker task plugin jar package from this dir.
taskPluginDir="lib/plugin/task"

# resource storage type: HDFS, S3, NONE（资源中心，比存储MR中使用的jar包）
resourceStorageType="HDFS"  # 这里我们存储在HDFS中

# resource store on HDFS/S3 path, resource file will store to this hdfs path, self configuration, please make sure the directory exists on hdfs and has read write permissions. "/dolphinscheduler" is recommended
# 存储资源的根目录
resourceUploadPath="/dolphinscheduler"

# if resourceStorageType is HDFS，defaultFS write namenode address，HA, you need to put core-site.xml and hdfs-site.xml in the conf directory.
# if S3，write S3 address，HA，for example ：s3a://dolphinscheduler，
# Note，S3 be sure to create the root directory /dolphinscheduler
# 默认FS得知（我们用hadoop的）
defaultFS="hdfs://master:9000"

# if resourceStorageType is S3, the following three configuration is required, otherwise please ignore
s3Endpoint="http://192.168.xx.xx:9010"
s3AccessKey="xxxxxxxxxx"
s3SecretKey="xxxxxxxxxx"

# resourcemanager port, the default value is 8088 if not specified
resourceManagerHttpAddressPort="8088"

# if resourcemanager HA is enabled, please set the HA IPs; if resourcemanager is single node, keep this value empty
# 启用了HA
yarnHaIps=""

# if resourcemanager HA is enabled or not use resourcemanager, please keep the default value; If resourcemanager is single node, you only need to replace 'yarnIp1' to actual resourcemanager hostname
# 未启用HA
singleYarnIp="master"

# who has permission to create directory under HDFS/S3 root path
# Note: if kerberos is enabled, please config hdfsRootUser=
hdfsRootUser="hdfs"

# kerberos config
# whether kerberos starts, if kerberos starts, following four items need to config, otherwise please ignore
# hdfs超级用户（谁启动的）
kerberosStartUp="false"
# kdc krb5 config file path
krb5ConfPath="$installPath/conf/krb5.conf"
# keytab username,watch out the @ sign should followd by \\
keytabUserName="hdfs-mycluster\\@ESZ.COM"
# username keytab path
keytabPath="$installPath/conf/hdfs.headless.keytab"
# kerberos expire time, the unit is hour
kerberosExpireTime="2"

# use sudo or not
sudoEnable="true"

# worker tenant auto create
workerTenantAutoCreate="true"
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178

3.5 一键部署 DolpinScheduler

1、启动 Zookeeper 集群

zh.sh start
1

2、一键部署并启动DolphinScheduler

./install.sh
1

3、查看 DolphinScheduler进程

jps
1

4、访问DophinScheduler UI

http://192.168.56.21:12345/dolphinscheduler
1

初始用户名：admin
初始密码：dolphinscheduler123

3.6 DolphinScheduler启停命令

进入到安装目录 cd /usr/local/ds/ds_install
1、一键启停所有服务

./bin/start-all.sh
./bin/stop-all.sh
1
2

2、启停 Master

./dolphinscheduler-daemon.sh start master-server
./dolphinscheduler-daemon.sh stop master-server
1
2

3、启停 Worker

./dolphinscheduler-daemon.sh start worker-server
./dolphinscheduler-daemon.sh stop worker-server
1
2

4、启停 Api

./dolphinscheduler-daemon.sh start api-server
./dolphinscheduler-daemon.sh stop api-server
1
2

5、启停 Logger

./dolphinscheduler-daemon.sh start logger-server
./dolphinscheduler-daemon.sh stop logger-server
1
2

4 DolphinScheduler入门

4.1 安全中心配置

4.1.1 Yarn队列管理

队列是在执行spark、mapreduce等程序，需要用到“队列”参数时使用的。
此处的队列对应的是Yarn调度器资源队列。故队列概念只对泡在Yarn上的任务类型有效。此处创建出的队列，可供后续任务进行选择。需要注意的是，在DolphinSheduler中创建队列，并不会影响到Yarn调度器的队列配置。

4.1.2 租户管理

租户对应的是 Linux的用户，用于worker提交作业所使用的用户。如果 linux 没有这个用户，默认情况下回导致任务运行失败。可以通过修改 worker.properties 配置文件中参数 worker.tenant.auto.create=true(默认值为flase)实现当linux用户不存在时自动创建该用户。
worker.tenant.auto.create=true 参数会要求 worker 可以免密运行 sudo 命令。

vim /usr/local/ds/ds_install/conf/worker.properties
1

报错处理：

org.apache.dolphinscheduler.registry.api.RegistryException: zookeeper connect timeout

vim registry.properties
1

registry.connection.timeout.ms=75000
1

org.apache.dolphinscheduler.api.exceptions.ApiExceptionHandler:[46] - 创建租户错误
查看

vim /usr/local/hadoop-2.8.4/etc/hadoop/core-site.xml
1

<!--配置hdfs默认的命名-->

	fs.defaultFS</name>
	hdfs://master:9000</value>
</property>
1
2
3
4
5

资源存储类型：HDFS,S3,NONE，修改 install_config.conf

如果resourceStorageType是HDFS，defaultFS写入namenode地址，HA需要把core-site.xml和hdfs-site.xml放到conf目录下。
如果是S3，写S3地址，HA，例如：s3a://dolphinscheduler，
注意，s3一定要创建根目录/dolphinscheduler
1
2
3

org.apache.hadoop.ipc.RemoteException: Permission denied: user=hdfs, access=WRITE, inode="/":root:supergroup:drwxr-xr-x

org.apache.hadoop.security.AccessControlException: Permission denied: user=hdfs, access=WRITE, inode="/":root:supergroup:drwxr-xr-x
对非CDH用户

vim /usr/local/hadoop-2.8.4/etc/hadoop/hdfs-site.xml
1

<!--是否启用hdfs的权限-->

    dfs.permissions</name>
    false</value>
</property>
1
2
3
4
5

参考地址：
https://blog.csdn.net/g11d111/article/details/72902112

参考地址：
https://www.bilibili.com/video/BV1sa411x7Ep
https://github.com/apache/dolphinscheduler/issues/4389