大数据——DBT：dbt集成数据质量监控插件elementary（生产环境篇）

官网地址：https://docs.elementary-data.com/introduction

1、准备
安装好dbt相关环境

2、集成
文件位置在dbt项目的根目录下

将以上文件替换为：

packages:
  - package: dbt-labs/dbt_utils
    version: [">=0.8.0", "<0.9.0"]

  - package: elementary-data/elementary
    version: 0.5.1
1
2
3
4
5
6

3、修改dbt配置文件
文件修改也可参考官网方法：elementary集成数据库配置

vim  /root/.dbt/profiles.yml
1

dbt_dags:
  outputs:
    dev:
      type: redshift
      threads: 1
      host: "{{ env_var('REDSHIFT_HOST') }}"
      cluster_id: "{{ env_var('REDSHIFT_CLUSTER_ID') }}"
      port: 5439
      #method: iam
      #pass: "93J#O3VEkel$SXVo"
      user: "{{ env_var('REDSHIFT_USER') }}"
      pass: "{{ env_var('REDSHIFT_PASSWD') }}"
      dbname: "{{ env_var('REDSHIFT_DBNAME') }}"
      schema: "{{ env_var('REDSHIFT_SCHEMA') }}"
  target: dev

elementary:
  outputs:
    dev:
      type: redshift
      threads: 1
      host: "{{ env_var('REDSHIFT_HOST') }}"
      cluster_id: "{{ env_var('REDSHIFT_CLUSTER_ID') }}"
      port: 5439
      #method: iam
      #pass: "93J#O3VEkel$SXVo"
      user: "{{ env_var('REDSHIFT_USER') }}"
      pass: "{{ env_var('REDSHIFT_PASSWD') }}"
      dbname: "{{ env_var('REDSHIFT_DBNAME') }}"
      schema: monitor
  target: dev
  
config:
    send_anonymous_usage_stats: False
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34

dbt项目根目录下：

vim dbt_project.yml
1

# Name your project! Project names should contain only lowercase characters
name: 'test_project'
version: '1.0.0'
config-version: 2

# This setting configures which "profile" dbt uses for this project.
profile: 'test_project'

model-paths: ["models"]
analysis-paths: ["analyses"]
test-paths: ["tests"]
docs-paths: ["docs"]
seed-paths: ["seeds"]
macro-paths: ["macros"]
snapshot-paths: ["snapshots"]

target-path: "target"  # directory which will store compiled SQL files
clean-targets:         # directories to be removed by `dbt clean`
  - "target"
  - "dbt_packages"
vars:
  # 公共变量
  dt: '{{ (modules.datetime.date.today()).strftime("%Y-%m-%d") }}'
  tests_schema_name: '_test'

models:
  +persist_docs:
    relation: true
    columns: true
  elementary:
    +schema: monitor
    internal:
      +schema: monitor
  dbt_dags:
    ods:
      schema: ods
    dwd:
      schema: dwd
    dws:
      schema: dws
    ads:
      schema: ads
    monitor:
      schema: monitor

seeds:
  dbt_dags:
    +schema: monitor
    
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49

4、关闭跟踪
在 HOME_DIR/.edr/config.yml 下新建目录和 yml 文件
默认情况下，这个完全匿名的跟踪是打开的。您可以随时选择退出，方法是将以下内容添加到您的config.yml文件中：

vim .edr/config.yml 
1

anonymous_usage_tracking: False
1

5、加载包

dbt deps
1

6、安装插件

pip install elementary-data
pip install 'elementary-data[redshift]'
1
2

7、测试安装

edr monitor --help
1

8、加载监控数据 Run to create the package models

dbt run --select elementary
1

or
重新创建基础表

dbt run -s elementary --full-refresh
1

elementary数据可以重新生成到数据库中

dbt run --select edr.dbt_artifacts
1

9、生成指定文件名称的可视化文档

edr monitor report --file-path dbt_edr_monitor_report_info.html
1