Filebeat自定义index和fields

如同logstash，默认会生成filebeat-*的索引以及使用解压目录下的fields.yml(二进制压缩包)

这里的需求是解析生成的特定文件，每行记录为json格式的数据，需要：

• 不同的文件生成不同的索引；
• 类型需要定义，原始查询中对于字符串未使用.keyword,因为默认模板使用的是小于1024是keyword,反之是text，如果要当作keyword使用则是fieldName.keyword;

配置

setup.template.name: custom_name
setup.template.pattern: custom_name_*
setup.template.enabled: false
setup.template.overwrite: false
setup.template.fields: customFields.yml
setup.ilm.enabled: false
processors:
- drop_fields:
    fields: [log, host, input, agent, ecs]
    ignore_missing: false
filebeat.inputs:
- close_removed: true
  close_inactive: 5m
  type: log
  tags: [t1]
  clean_removed: true
  enabled: true
  json: {keys_under_root: true, overwrite_keys: true}
  paths: [/data/t1_*]
- close_removed: true
  close_inactive: 5m
  type: log
  tags: [t2]
  clean_removed: true
  enabled: true
  json: {keys_under_root: true, overwrite_keys: true}
  paths: [/data/t2_*]
output.elasticsearch:
  indices:
  - index: custom_name_t1
    when.contains: {tags: t1}
  - when.contains: {tags: t2}
    index: custom_name_t2
  hosts: ['127.0.0.1:9200']

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35

setup.template.name 设置一个新的模板，模板的名称
setup.template.pattern 模板匹配那些索引
setup.template.enabled: false 关掉默认的模板配置
setup.template.overwrite: false 是否覆盖现有模板
when.contains: 包含
keys_under_root: true 将field展开到最外层(the custom fields are stored as top-level fields in the output document)

自定义模板请查看：Configuration-template

文件路径请查看：Configure project paths

字段定义

setup.template.fields: customFields.yml(路径请查看上述链接)，文件格式如下：

- key: custom_name
  title: custom_name
  description: >
    custom fields
  fields:
  # some desc
    - name: t1
      type: keyword
    - name: t2
      type: keyword
    - name: t3
      type: ip
    - name: t4
      type: integer
    - name: t5
      type: ip
    - name: t6
      type: integer
    - name: t7
      type: ip
    - name: t8
      type: text
    - name: t9
      type: date
    - name: t10
      type: long
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26