ELK之LogStash插件grok和geoip的配置使用

本文针对LogStash常用插件grok和geoip的使用进行说明：

一、使用grok输出结构化数据

编辑 first-pipeline.conf 文件，修改为如下内容：

input{
  #stdin{type => stdin}
  file {
    # 读取文件的路径
    path => ["/tmp/access.log"]
    start_position => "beginning"
  }
}

filter{
  grok{
    match => {"message" => "%{COMBINEDAPACHELOG}" }
  }

}

output{
  stdout{codec => rubydebug}
}

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20

启动./logstash -f ../config/first-pipeline.conf后输出就为结构化的数据了：

{
        "message" => "140.77.188.102 - - [25/Jun/2022:05:11:33 +0800] \"GET /api/ss/api/v1/login/getBaseUrl HTTP/1.1\" 200 103 \"-\" \"Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/534+ (KHTML, like Gecko) BingPreview/1.0b\"",
       "response" => "200",
           "auth" => "-",
          "bytes" => "103",
       "referrer" => "\"-\"",
           "host" => "nb002",
       "@version" => "1",
          "agent" => "\"Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/534+ (KHTML, like Gecko) BingPreview/1.0b\"",
     "@timestamp" => 2022-06-26T00:28:24.302Z,
      "timestamp" => "25/Jun/2022:05:11:33 +0800",
          "ident" => "-",
    "httpversion" => "1.1",
           "path" => "/tmp/access.log",
       "clientip" => "140.77.188.102",
           "verb" => "GET",
        "request" => "/api/ss/api/v1/login/getBaseUrl"
}
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18

二、使用grok对输出数据进行修改

编辑 first-pipeline.conf 文件，修改为如下内容：

input{
  #stdin{type => stdin}
  file {
    path => ["/tmp/access.log"]
    start_position => "beginning"
  }
}

filter{
  grok{
    match => {"message" => "%{COMBINEDAPACHELOG}" }
  }
  mutate{
    # 重命名字段
    rename => {"clientip" => "cip"}
  }
  mutate{
    # 移出特定字段
    remove_field => ["timestamp","agent"]
  }
}

output{
  stdout{codec => rubydebug}
}

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26

重新启动./logstash -f ../config/first-pipeline.conf 后，往 /tmp/access.log 中新增一条数据，看输出：发现"clientip" 变成了 “cip” 和timestamp agent 字段已经没有了。NICE

{
           "verb" => "GET",
     "@timestamp" => 2022-06-26T00:48:28.224Z,
       "referrer" => "\"-\"",
           "path" => "/tmp/access.log",
           "auth" => "-",
        "message" => "140.77.188.102 - - [25/Jun/2022:05:11:33 +0800] \"GET /api/ss/api/v1/login/getBaseUrl HTTP/1.1\" 200 103 \"-\" \"Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/534+ (KHTML, like Gecko) BingPreview/1.0b\"",
       "@version" => "1",
          "ident" => "-",
       "response" => "200",
          "bytes" => "103",
        "request" => "/api/ss/api/v1/login/getBaseUrl",
    "httpversion" => "1.1",
           "host" => "nb002",
            "cip" => "140.77.188.102"
}

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17

三、使用geoip过滤器插件

使用geoip过滤器插件，可以增强数据。
geoip插件可以针对IP地址进行地理位置信息来源的查找

编辑 first-pipeline.conf 文件，修改为如下内容：

input{
  #stdin{type => stdin}
  file {
    path => ["/tmp/access.log"]
    start_position => "beginning"
  }
}

filter{
  grok{
    match => {"message" => "%{COMBINEDAPACHELOG}" }
  }
  mutate{
    # 重命名字段
    rename => {"clientip" => "cip"}
  }
  mutate{
    # 移出特定字段
    remove_field => ["timestamp","agent"]
  }
  geoip{
    # 由于上面将clientip修改为了cip，故此处配置cip，如果没有rename字段则用clientip
    source => "cip"
  }
}

output{
  stdout{codec => rubydebug}
}
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29

重新启动./logstash -f ../config/first-pipeline.conf 后，往 /tmp/access.log 中新增一条数据，看输出：发现输出结果中新增了geoip 字段，并展示了地区、国家、省份、经纬度等地理位置信息。

外国ip示例：

{
           "host" => "nb002",
           "auth" => "-",
          "bytes" => "103",
            "cip" => "140.77.188.104",
       "@version" => "1",
        "message" => "140.77.188.104 - - [25/Jun/2022:05:11:33 +0800] \"GET /api/ss/api/v1/login/getBaseUrl HTTP/1.1\" 200 103 \"-\" \"Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/534+ (KHTML, like Gecko) BingPreview/1.0b\"",
           "verb" => "GET",
        "request" => "/api/ss/api/v1/login/getBaseUrl",
       "referrer" => "\"-\"",
       "response" => "200",
          "ident" => "-",
           "path" => "/tmp/access.log",
     "@timestamp" => 2022-06-26T00:58:11.786Z,
          "geoip" => {
	         "country_code3" => "FR",
	             "longitude" => 4.85,
	                    "ip" => "140.77.188.104",
	        "continent_code" => "EU",
	           "region_name" => "Rhône",
	         "country_code2" => "FR",
	              "timezone" => "Europe/Paris",
	          "country_name" => "France",
	           "region_code" => "69",
	              "latitude" => 45.748,
	           "postal_code" => "69007",
	              "location" => {
	            "lat" => 45.748,
	            "lon" => 4.85
        },
             "city_name" => "Lyon"
    },
    "httpversion" => "1.1"
}

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35

国内ip示例：

{
           "host" => "nb002",
           "auth" => "-",
          "bytes" => "103",
            "cip" => "175.30.108.241",
       "@version" => "1",
        "message" => "175.30.108.241 - - [25/Jun/2022:05:11:33 +0800] \"GET /api/ss/api/v1/login/getBaseUrl HTTP/1.1\" 200 103 \"-\" \"Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/534+ (KHTML, like Gecko) BingPreview/1.0b\"",
           "verb" => "GET",
        "request" => "/api/ss/api/v1/login/getBaseUrl",
       "referrer" => "\"-\"",
       "response" => "200",
          "ident" => "-",
           "path" => "/tmp/access.log",
     "@timestamp" => 2022-06-26T01:00:11.972Z,
          "geoip" => {
         "country_code3" => "CN",
             "longitude" => 125.3247,
                    "ip" => "175.30.108.241",
        "continent_code" => "AS",
           "region_name" => "Jilin",
         "country_code2" => "CN",
              "timezone" => "Asia/Shanghai",
          "country_name" => "China",
           "region_code" => "JL",
              "latitude" => 43.88,
              "location" => {
            "lat" => 43.88,
            "lon" => 125.3247
        },
             "city_name" => "Changchun"
    },
    "httpversion" => "1.1"
}

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34

END