graylog日志部署与使用

简介

graylog是一个简单易用、功能较全面的日志管理工具，graylog也采用Elasticsearch作为存储和索引以保障性能，MongoDB用来存储少量的自身配置信息，master-node模式具有很好的扩展性，UI上自带的基础查询与分析功能比较实用且高效，支持LDAP、权限控制并有丰富的日志类型和标准（如syslog，GELF）并支持基于日志的报警。

docker部署

docker-compose.yml:

version: "3"
services:

  mongo:
    image: mongo:4.2
    command: --bind_ip 0.0.0.0
    ports:
      - "27700:27017"
    volumes:
      - /data/gray_log_test/mongo_data:/data/db
      - 
  elasticsearch:
    image: secureimages/elasticsearch-oss:7.10.2-alpine-3.13.2
    environment:
      - http.host=0.0.0.0
      - transport.host=localhost
      - network.host=0.0.0.0
    ulimits:
      memlock:
        soft: -1
        hard: -1
    deploy:
      resources:
        limits:
          memory: 1g

  graylog:
    image: graylog/graylog:4.2
    user: root
    env_file:
      - docker.env
    entrypoint: /usr/bin/tini -- wait-for-it elasticsearch:9200 --  /docker-entrypoint.sh
    depends_on:
      - mongo
      - elasticsearch
    ports:
      - "9001:9000"
      - "12201:12201"
      - "12201:12201/udp"
    volumes:
      - /data/gray_log_test/graylog_data:/usr/share/graylog/data
      - ./graylog.conf:/usr/share/graylog/data/config/graylog.conf
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42

docker.env:

GRAYLOG_PASSWORD_SECRET=xxxxxxx
GRAYLOG_ROOT_PASSWORD_SHA2=xxxxxxx

TZ=Asia/Shanghai
ROOT_TIMEZONE=Asia/Shanghai
GRAYLOG_TIMEZONE=Asia/Shanghai
1
2
3
4
5
6

graylog.conf

is_master = true
node_id_file = /usr/share/graylog/data/config/node-id
root_timezone = Asia/Shanghai
bin_dir = /usr/share/graylog/bin
data_dir = /usr/share/graylog/data
plugin_dir = /usr/share/graylog/plugin
http_bind_address = 0.0.0.0:9000
elasticsearch_hosts = http://elasticsearch:9200
elasticsearch_max_docs_per_index = 20000000
elasticsearch_max_number_of_indices = 20
elasticsearch_shards = 4
elasticsearch_replicas = 0
elasticsearch_index_prefix = graylog
allow_leading_wildcard_searches = false
allow_highlighting = false
rotation_strategy = count
output_batch_size = 500
output_flush_interval = 1
output_fault_count_threshold = 5
output_fault_penalty_seconds = 30
processbuffer_processors = 5
outputbuffer_processors = 3
ring_size = 65536
inputbuffer_ring_size = 65536
inputbuffer_processors = 2
inputbuffer_wait_strategy = blocking
message_journal_enabled = true
message_journal_dir = data/journal
lb_recognition_period_seconds = 3
mongodb_uri = mongodb://mongo/graylog
mongodb_max_connections = 1000
mongodb_threads_allowed_to_block_multiplier = 5
proxied_requests_thread_pool_size = 32
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33

使用docker-compose up -d 命令启动容器

使用

在浏览器中输入http://127.0.0.1:9001/, 用户名admin, 密码是自己文件配置的

点击system->inputs, 选择select input->GELF HTTP（GELF UDP）->lanch new input,端口都选择在12201

python写日志

python版本：python3.7.13
输入命令:

 pip install graypy
1

新建demo.py

import logging
import graypy
import time
my_logger = logging.getLogger('test_logger')
my_logger.setLevel(logging.DEBUG)
formatter = logging.Formatter(
    "%(asctime)s - %(threadName)s - %(levelname)s - %(pathname)s.%(funcName)s line %(lineno)d - %(message)s")

handler = graypy.GELFUDPHandler('127.0.0.1', 12201)
handler.setFormatter(formatter)
my_logger.addHandler(handler)

for i in range(1000):
    my_logger.warning(f'type:udp,index:{i}')

for i in range(1000):
    my_logger.warning(f'type:tcp,index:{i}')
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17

查询

点击search,输入"type:udp"，点击搜索，就可以查看到结果

var code = "df968d80-7b2b-43cb-a734-a36ce1c4be9a"
1