logback+MQ+Logstash 日志收集

需求场景

采集多台服务（下文用生产端代替）的日志数据，汇总到一台服务器（下文用消费端代替）中。

实现

1.RabbitMQ 将日志消息发布到mq，消费端获取消息。
2.Logstash 收集日志数据。放在消费端，接收mq消息，文件形式输出到本地。

服务环境

1.windows环境
2 .mq（环境模拟）

生产端配置

1.在logback.xml中配置MQ

1.引入依赖

<dependency>
<groupId>org.springframework.bootgroupId>
<artifactId>spring-boot-starter-amqpartifactId>
dependency>
<dependency>
<groupId>org.springframework.amqpgroupId>
<artifactId>spring-rabbitartifactId>
dependency>
1
2
3
4
5
6
7
8

2.logback.xml 修改
以一个为例，分别在项目中配置即可

   <appender name="RABBITMQ" class="org.springframework.amqp.rabbit.logback.AmqpAppender">
        <layout>
            <pattern>%d{yyyy-MM-dd HH:mm:ss.SSS} [%thread] %X{requestId}] %-5level %logger{50} - %msg%npattern>
        layout>
        <host>192.168.1.209host>
        <port>5672port>
        <username>guestusername>
        <password>guestpassword>
        <declareExchange>truedeclareExchange>
        <exchangeType>directexchangeType>
        <exchangeName>app.logexchangeName>
        <routingKeyPattern>inforoutingKeyPattern>
        <generateId>truegenerateId>
        <charset>UTF-8charset>
        <durable>truedurable>
        <deliveryMode>NON_PERSISTENTdeliveryMode>
        <autoDelete>falseautoDelete>
    appender>
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18

root中配置appender-ref

    <root level="info">
        <appender-ref ref="RABBITMQ" />
    root>
1
2
3

消费端配置

1.下载erlang，rabbitmq 安装自行百度，一步到底，可视化需要安装插件，百度一大堆
2.下载logstash运行

logstash.conf 配置

以下以收集两个服务的日志为例

input {
 
  rabbitmq {
       type =>"app"
       durable => true
       exchange => "app.log"
       exchange_type => "direct"
       key => "info"
       host => "192.168.1.209"
       port => 5672
       user => "guest"
       password => "guest"
       queue => "log_queue"
       auto_delete => false
  }
  
    rabbitmq {
       type =>"fmis"
       durable => true
       exchange => "fmis.log"
       exchange_type => "direct"
       key => "info"
       host => "192.168.1.209"
       port => 5672
       user => "guest"
       password => "guest"
       queue => "log_queue"
       auto_delete => false
  }
 
}
 

output {
 
   if [type] == "app" {
    file {
      path => "E:/test-log/app-%{+YYYY-MM-dd}.log"
      codec => multiline {
            pattern => "^\d"
            negate => true
            what => "previous"
        }
 
    }
  }
   if [type] == "fmis" {
    file {
      path => "E:/test-log/fmis-%{+YYYY-MM-dd}.log"
      codec => multiline {
            pattern => "^\d"
            negate => true
            what => "previous"
        }
 
    }
  }
}
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58

配置好后，启动rabitmq和logstash

效果

访问 192.168.1.209:15672

2.查看本地