dolphinscheduler 3.0.1数据质量

🔼上一集：dolphinscheduler 3.0.1 资源中心

*️⃣主目录：dolphinscheduler 3.0.1功能梳理及源码解读

🔽下一集：dolphinscheduler 3.0.1 数据源中心

🐬环境配置

---

• /worker-server/conf/dolphinscheduler_env.sh
  • SPARK_HOME2 ：配置spark安装目录
  • HADOOP_USER_NAME：增加该变量，填写hadoop集群的部署用户(HADOOP_HOME等不需要配置，目前我是没遇到过)
    

    org.apache.hadoop.security.AccessControlException): Permission denied: user=dolphin, access=WRITE
    1

• /worker-server/conf/common.properties
  • data-quality.jar.name=dolphinscheduler-data-quality-3.0.1-SNAPSHOT.jar :保持jar包名称和编译后的名称一致，默认为dolphinscheduler-data-quality-dev-SNAPSHOT.jar
• 给执行租户赋权，添加hadoop部署用户组，比如租户dolphin,hadoop部署用户组为bigdata

  sudo usermod -a -G bigdata dolphin
  1

🐬定义质量节点测试

---

直接对着官网操作吧，最开始官网资料还没有3.0，后来就不看了，貌似多走了一些路。

• 校验公式：[校验方式][操作符][阈值]，如果结果为真，则表明数据不符合期望，执行失败策略
• 校验方式：
  • [Expected-Actual][期望值-实际值] [Actual-Expected][实际值-期望值]
  • [Actual/Expected][实际值/期望值]x100%
  • [(Expected-Actual)/Expected][(期望值-实际值)/期望值]x100%
• 操作符：=、>、>=、<、<=、!=
• 期望值类型： 固定值 日均值 周均值 月均值 最近7天均值 最近30天均值 源表总行数 目标表总行数
• 例子：
  • 校验方式为：[Expected-Actual][期望值-实际值] [操作符]：> [阈值]：0 期望值类型：固定值=9。
  • 假设实际值为10，操作符为 >, 期望值为9，那么结果 10 -9 > 0 为真，那就意味列为空的行数据已经超过阈值，任务被判定为失败
    

🐠运行失败

---

按理说应该成功才对，结果不会小于0的

错误信息：

[INFO] 2022-10-26 08:20:10.909 +0000 [taskAppId=TASK-20221026-7339496693088_2-471-499] TaskLogLogger-class org.apache.dolphinscheduler.plugin.task.dq.DataQualityTask:[205] - process has exited, execute path:/tmp/dolphinscheduler3/exec/process/7338799615584/7339496693088_2/471/499, processId:260177 ,exitStatusCode:1 ,processWaitForStatus:true ,processExitValue:1
[INFO] 2022-10-26 08:20:11.582 +0000 [taskAppId=TASK-20221026-7339496693088_2-471-499] TaskLogLogger-class org.apache.dolphinscheduler.plugin.task.dq.DataQualityTask:[63] -  -> 22/10/26 16:20:10 INFO Client: Application report for application_1658989604268_0102 (state: FAILED)
	22/10/26 16:20:10 INFO Client: 
		 client token: N/A
		 diagnostics: Application application_1658989604268_0102 failed 1 times (global limit =2; local limit is =1) due to ApplicationMaster for attempt appattempt_1658989604268_0102_000001 timed out. Failing the application.
		 ApplicationMaster host: N/A
		 ApplicationMaster RPC port: -1
		 queue: default
		 start time: 1666771787933
		 final status: FAILED
		 tracking URL: http://bigdata02:8088/cluster/app/application_1658989604268_0102
		 user: hadoop
	22/10/26 16:20:10 ERROR Client: Application diagnostics message: Application application_1658989604268_0102 failed 1 times (global limit =2; local limit is =1) due to ApplicationMaster for attempt appattempt_1658989604268_0102_000001 timed out. Failing the application.
	Exception in thread "main" org.apache.spark.SparkException: Application application_1658989604268_0102 finished with failed status
		at org.apache.spark.deploy.yarn.Client.run(Client.scala:1283)
		at org.apache.spark.deploy.yarn.YarnClusterApplication.start(Client.scala:1677)
		at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:955)
		at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:180)
		at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:203)
		at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:90)
		at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1043)
		at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1052)
		at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
	22/10/26 16:20:10 INFO ShutdownHookManager: Shutdown hook called
	22/10/26 16:20:10 INFO ShutdownHookManager: Deleting directory /tmp/spark-869c7fa1-4136-4458-8626-27961ce772e1
	22/10/26 16:20:10 INFO ShutdownHookManager: Deleting directory /tmp/spark-873afb0a-8352-48c7-981a-7be426e1c145
[INFO] 2022-10-26 08:20:11.582 +0000 [taskAppId=TASK-20221026-7339496693088_2-471-499] TaskLogLogger-class org.apache.dolphinscheduler.plugin.task.dq.DataQualityTask:[57] - FINALIZE_SESSION
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27

测试用的spark是3.2版本，不知道报错是不是这个原因，暂时没时间研究了，有知道的欢迎指教

🐬源码

---

简单过了下，不深究了，后面还会单独研究后台核心代码

其它(spark-submit)20221114补充

spark-submit官网

组装好的命令没有看到--class配置，因为数据质量jar包中已有main函数，会直接调用该Main-Class: org.apache.dolphinscheduler.data.quality.DataQualityApplication

${SPARK_HOME2}/bin/spark-submit --master yarn --deploy-mode cluster --driver-cores 1 --driver-memory 512M --num-executors 2 --executor-cores 2 --executor-memory 2G --queue default --conf spark.yarn.maxAppAttempts=1 /home/dolphinscheduler/app/dolphinscheduler/worker-server/libs/dolphinscheduler-data-quality-3.0.1-SNAPSHOT.jar
1

jar包设置了Main-Class