ELK单机版部署踩坑及与Springboot整合

部署ELK单机版，要将所有的Springboot业务应用的日志对接ELK。记录下详细的部署过程，以及一个问题的排查记录。

这张图比较好的反映出了三个组件之间的关系，下面的配置过程也是配这些组件两两的关系。

部署环境

服务器：CentOS7.8，4核32GB，IP：192.188.1.246

elk三个软件版本要一致：

elasticsearch：elasticsearch-7.9.3-x86_64.rpm

下载地址

logstash：kibana-7.9.3-x86_64.rpm

下载地址

kibina：kibana-7.9.3-x86_64.rpm

下载地址

部署流程

(1)关闭selinux,设置主机名

root用户登录，关闭防火墙
# hostnamectl set-hostname elk-server

(2)安装JDK，elasticsearch ，logstash，kibana
# yum -y install java-1.8.0-openjdk*
# yum -y install elasticsearch-7.9.3-x86_64.rpm
# yum -y install kibana-7.9.3-x86_64.rpm
# yum -y install logstash-7.9.3-x86_64.rpm

(3)修改elasticsearch配置文件

vi /etc/elasticsearch/elasticsearch.yml

cluster.name: elk-server  (集群名字，ELK单机版可以随便取)
node.name: elk-server   (ELK本机主机名)
path.data: /home/elk/elasticsearch
path.logs: /var/log/elasticsearch
network.host: 192.188.1.246  (ELK本机IP)
http.port: 9200
discovery.seed_hosts: ["elk-server"]   (ELK本机主机名)
cluster.initial_master_nodes: ["192.188.1.246"]  (ELK本机IP)

(3)配置路径及用户

# mkdir -p /home/elk/elasticsearch

# chown -R elasticsearch:elasticsearch /home/elk/elasticsearch

(5)启动elasticsearch
# systemctl start elasticsearch
# systemctl enable elasticsearch
# systemctl status elasticsearch

(6)修改logstash配置文件
# vim /etc/logstash/logstash.yml

node.name: elk-server   (ELK本机主机名)
path.data: /home/elk/logstash  (logstash数据存储路径)
pipeline.ordered: auto
path.config: /etc/logstash/conf.d (配置文件路径)
log.level: info
path.logs: /var/log/logstash
xpack.monitoring.enabled: true
xpack.monitoring.elasticsearch.hosts: ["http://192.188.1.246:9200"]

(7)修改kibana配置文件
# vim /etc/kibana/kibana.yml

server.port: 5601
server.host: "192.188.1.246"
server.name: "elk-server"
elasticsearch.hosts: ["http://192.188.1.246:9200"]  (kibana连接elasticsearch的地址)
kibana.index: ".kibana"
i18n.locale: "zh-CN"

(8)设置logstash的输入输出

# vim /etc/logstash/conf.d/apps.conf （文件名可任意）

#输入在5044端口侦听TCP连接
input{
  tcp {
    host => "192.188.1.246"
    port => 5044
    codec => json_lines
  }
}
#输出到es中
output{
  elasticsearch{
    hosts => ["192.188.1.246:9200"]
    index => "applog"
  }
}

(9)启动kibana和logstash
[root@elk-log-server ~]# systemctl start kibana
[root@elk-log-server ~]# systemctl enable kibana
[root@elk-log-server ~]# systemctl status kibana
[root@elk-log-server ~]# systemctl start logstash
[root@elk-log-server ~]# systemctl enable logstash
[root@elk-log-server ~]# systemctl status logstash

elk单机环境搭建完成，可用浏览器访问：http://1192.188.1.246:5601

这时候，ES中还没有任何数据，添加不了索引模式。

 与Springboot整合

pom文件：

<dependency>
    <groupId>net.logstash.logbackgroupId>
    <artifactId>logstash-logback-encoderartifactId>
    <version>5.3version>
dependency>

resources/logback-spring.xml文件中添加logstash的配置

"1.0" encoding="UTF-8"?>
<configuration debug="false" scan="true" scanPeriod="10 seconds">
 
    
    <include resource="org/springframework/boot/logging/logback/defaults.xml" />
    <include resource="org/springframework/boot/logging/logback/console-appender.xml" />
    <springProperty scope="context" name="file_basePath" source="logging.file_basePath" defaultValue="./logs"/>
    <springProperty scope="context" name="file_prefix" source="logging.file_prefix" defaultValue="application"/>
 
    
    <appender name="file" class="ch.qos.logback.core.rolling.RollingFileAppender">
        
        
 
        
        <append>trueappend>
        
        <rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy">
            <fileNamePattern>${file_basePath}/${file_prefix}/${file_prefix}-%d{yyyy-MM-dd}.%i.logfileNamePattern>
            
            <maxHistory>30maxHistory>
            
            <timeBasedFileNamingAndTriggeringPolicy class="ch.qos.logback.core.rolling.SizeAndTimeBasedFNATP">
                
                <maxFileSize>50MBmaxFileSize>
            timeBasedFileNamingAndTriggeringPolicy>
        rollingPolicy>
 
        
        <encoder>
            <pattern>%d{yyyy-MM-dd HH:mm:ss.SSS} [%thread] %-5level %logger - %msg%npattern>
        encoder>
    appender>
 
    
    <appender name="LOGSTASH" class="net.logstash.logback.appender.LogstashTcpSocketAppender">
        <destination>192.188.1.246:5044destination>
        
        <encoder charset="UTF-8" class="net.logstash.logback.encoder.LoggingEventCompositeJsonEncoder">
            <providers>
                <timestamp>
                    <timeZone>UTCtimeZone>
                timestamp>
                <pattern>
                    <pattern>
                        {
                        "logLevel": "%level",
                        "serviceName": "${springAppName:-}",
                        "pid": "${PID:-}",
                        "thread": "%thread",
                        "class": "%logger{40}",
                        "rest": "%message"
                        }
                    pattern>
                pattern>
            providers>
        encoder>
        
    appender>
 
    
    <root level="info">
        <appender-ref ref="CONSOLE" />
        <appender-ref ref="file" />
        <appender-ref ref="LOGSTASH" />
    root>
configuration>

踩坑

程序中打上几个log.info()，发送一些日志，再到网页上添加索引，还是没有数据。

打开http://192.188.1.246:9200/，有东西，表示es正常，

打开http://192.188.1.246:9200/_cat/indices?v

 没有logstash配置文件/etc/logstash/conf.d/apps.conf里面，配置的applog这个索引。

再打开logstash的日志/var/log/logstash，目录下的logstash-plain.log文件

 提示，/home/elk/logstash没有写入权限，尝试改变读写权限

chown -R logstash:logstash/home/elk/logstash

 再发送几个日志，此时可在kibana页面中添加索引模式

在Discovery页面可以看到日志了

 可以通了，后面就是详细的日志查看页面的配置了。