1.简介

参考我的专栏《ELK》

2.版本选择

• 查看 elasticsearch官网
  • 稳定版 7.17.1 【选择】
  • 版本不稳定 8.1.1

选择稳定版就可以

3.安装（Linux）

3.1.下载

官网下载最新包，参考本文【附录】

3.2.解压

tar -xzf elasticsearch-7.17.1-linux-x86_64.tar.gz
tar -xzf kibana-7.17.1-linux-x86_64.tar.gz
tar -xzf logstash-7.17.1-linux-x86_64.tar.gz
1
2
3

3.3.创建用户组

启动elasticsearch和kibana要求以非root用户启动

# root用户
groupadd es
useradd -g es es
chown -R es:es /opt/elk/*
# 设置es密码123456
passwd es
1
2
3
4
5
6

3.4.设置环境变量

vim /etc/profile

export ES_JAVA_HOME=/opt/elk/elasticsearch-7.17.1/jdk
export LS_JAVA_HOME=/opt/elk/logstash-7.17.1/jdk
export PATH=$ES_JAVA_HOME/bin:$LS_JAVA_HOME/bin:$PATH
source /etc/profile
1
2
3
4
5
6

3.5.修改最大操作文件数

vim /etc/security/limits.conf
* hard nofile 65536
* soft nofile 65536

vim /etc/sysctl.conf 
#添加 vm.max_map_count=655360,然后加载参数
#刷新配置文件
sysctl -p 
1
2
3
4
5
6
7
8

3.6.开放服务器端口

firewall-cmd --zone=public --add-port=9200/tcp --permanent 
firewall-cmd --zone=public --add-port=5601/tcp --permanent
firewall-cmd --zone=public --add-port=5044/tcp --permanent
firewall-cmd --reload
1
2
3
4

3.7.新建文件夹并授权

mkdir -pv /opt/elk/elasticsearch-7.17.1/{data,logs}
mkdir -pv /opt/elk/kibana-7.17.1/{data,logs}
mkdir -pv /opt/elk/logstash-7.17.1/{data,logs}

chown -R es:es /opt/elk/*
1
2
3
4
5

3.8.elasticsearch

1）生成密钥证书

参考

cd /opt/elk/elasticsearch-7.17.1/bin
sh elasticsearch-certutil ca
sh elasticsearch-certutil cert --ca elastic-stack-ca.p12
# 输入密码 123456
# 将生成的证书 放到 /opt/elk/elasticsearch-7.17.1/config
# elastic-certificates.p12
# elastic-stack-ca.p12
sh elasticsearch-keystore add xpack.security.transport.ssl.keystore.secure_password
sh elasticsearch-keystore add xpack.security.transport.ssl.truststore.secure_password
1
2
3
4
5
6
7
8
9

2）查看elk用户权限

vim /etc/passwd
es:x:1001:1001::/home/es:/bin/bash
1
2

3）配置

vim /opt/elk/elasticsearch-7.17.1/config/elasticsearch.yml

# 集群名称
cluster.name: es1-application
# 节点名称
node.name: node-1
# data存储目录
path.data: /opt/elk/elasticsearch-7.17.1/data
# logs日志的目录
path.logs: /opt/elk/elasticsearch-7.17.1/logs
# 允许所有ip连接
network.host: 0.0.0.0
# 监听的端口
http.port: 9200
cluster.initial_master_nodes: ["node-1"]
xpack.security.enabled: true
# xpack安全
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate 
xpack.security.transport.ssl.client_authentication: required
xpack.security.transport.ssl.keystore.path: elastic-certificates.p12
xpack.security.transport.ssl.truststore.path: elastic-certificates.p12

# https://blog.csdn.net/weixin_43820556/article/details/120165948
# 解决 [ERROR] exception during geoip databases update
# ingest.geoip.downloader.enabled: false
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26

4）启动

# 启动
sh /opt/elk/elasticsearch-7.17.1/bin/elasticsearch -d
# 查看是否运行
ps -ef | grep elastic
# 初始化内置用户密码
sh /opt/elk/elasticsearch-7.17.1/bin/elasticsearch-setup-passwords interactive
# 密码全部设置123456
1
2
3
4
5
6
7

5）重启ES

ps -ef | grep elastic
# xxx 为输出的第二列
kill -9 xxx
sh /opt/elk/elasticsearch-7.17.1/bin/elasticsearch -d
1
2
3
4

6）查看ES运行状态

curl http://IP:9200 -u elastic:123456
1

3.9.kibana

1）配置

vim /opt/elk/kibana-7.17.1/config/kibana.yml
server.port: 5601
server.host: "192.xxx.xxx.134"
elasticsearch.hosts: ["http://192.xxx.xxx.134:9200"]
kibana.index: ".kibana"
kibana.defaultAppId: "home"
elasticsearch.username: "kibana_system"
elasticsearch.password: "123456"
pid.file: /opt/elk/kibana-7.17.1/run/kibana/kibana.pid
logging.dest: /opt/elk/kibana-7.17.1/logs/stdout.log
i18n.locale: "zh-CN"
server.publicBaseUrl: "http://192.xxx.xxx.134:5601"
1
2
3
4
5
6
7
8
9
10
11
12

2）启动

nohup /opt/elk/kibana-7.17.1/bin/kibana &
ps -ef | grep kibana
1
2

3）查看服务状态

访问
http://192.xxx.xxx.135:5601
elastic:123456
1
2
3

4）问题

server.publicBaseUrl 缺失，在生产环境中运行时应配置。某些功能可能运行不正常。 请参阅文档。
1

3.10.logstash

待测试

1）配置

vim /opt/elk/logstash-7.17.1/config/logstash.yml

node.name: logstash_node1
path.data: /opt/elk/logstash-7.17.1/data
pipeline.id: logstash_node1
pipeline.workers: 2
pipeline.batch.size: 125
pipeline.batch.delay: 50
pipeline.ordered: auto
# ------------ API Settings -------------
api.enabled: true
api.http.host: 192.xxx.xxx.135
api.http.port: 9600-9700
api.environment: "production"
api.ssl.enabled: true
api.ssl.keystore.path: /path/to/keystore.jks
api.ssl.keystore.password: "123456"
api.auth.type: basic
api.auth.basic.username: "logstash-user"
api.auth.basic.password: "123456"

path.logs: /opt/elk/logstash-7.17.1/logs

# ------------ X-Pack Settings (not applicable for OSS build)--------------
#
# X-Pack Monitoring
# https://www.elastic.co/guide/en/logstash/current/monitoring-logstash.html
xpack.monitoring.enabled: true
xpack.monitoring.elasticsearch.username: logstash_system
xpack.monitoring.elasticsearch.password: "123456"
#xpack.monitoring.elasticsearch.proxy: ["http://proxy:port"]
xpack.monitoring.elasticsearch.hosts: ["https://192.xxx.xxx.134:9200"]
# an alternative to hosts + username/password settings is to use cloud_id/cloud_auth
#xpack.monitoring.elasticsearch.cloud_id: monitoring_cluster_id:xxxxxxxxxx
#xpack.monitoring.elasticsearch.cloud_auth: logstash_system:password
# another authentication alternative is to use an Elasticsearch API key
#xpack.monitoring.elasticsearch.api_key: "id:api_key"
#xpack.monitoring.elasticsearch.ssl.certificate_authority: [ "/path/to/ca.crt" ]
#xpack.monitoring.elasticsearch.ssl.truststore.path: path/to/file
#xpack.monitoring.elasticsearch.ssl.truststore.password: password
#xpack.monitoring.elasticsearch.ssl.keystore.path: /path/to/file
#xpack.monitoring.elasticsearch.ssl.keystore.password: password
xpack.monitoring.elasticsearch.ssl.verification_mode: certificate
#xpack.monitoring.elasticsearch.sniffing: false
#xpack.monitoring.collection.interval: 10s
#xpack.monitoring.collection.pipeline.details.enabled: true
#
# X-Pack Management
# https://www.elastic.co/guide/en/logstash/current/logstash-centralized-pipeline-management.html
#xpack.management.enabled: false
#xpack.management.pipeline.id: ["main", "apache_logs"]
#xpack.management.elasticsearch.username: logstash_admin_user
#xpack.management.elasticsearch.password: password
#xpack.management.elasticsearch.proxy: ["http://proxy:port"]
#xpack.management.elasticsearch.hosts: ["https://es1:9200", "https://es2:9200"]
# an alternative to hosts + username/password settings is to use cloud_id/cloud_auth
#xpack.management.elasticsearch.cloud_id: management_cluster_id:xxxxxxxxxx
#xpack.management.elasticsearch.cloud_auth: logstash_admin_user:password
# another authentication alternative is to use an Elasticsearch API key
#xpack.management.elasticsearch.api_key: "id:api_key"
#xpack.management.elasticsearch.ssl.certificate_authority: [ "/path/to/ca.crt" ]
#xpack.management.elasticsearch.ssl.truststore.path: /path/to/file
#xpack.management.elasticsearch.ssl.truststore.password: password
#xpack.management.elasticsearch.ssl.keystore.path: /path/to/file
#xpack.management.elasticsearch.ssl.keystore.password: password
#xpack.management.elasticsearch.ssl.verification_mode: certificate
#xpack.management.elasticsearch.sniffing: false
#xpack.management.logstash.poll_interval: 5s
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68

vim /opt/elk/logstash-7.17.1/config/logstash-sample.conf

# Beats -> Logstash -> Elasticsearch pipeline.

input {
  beats {
    port => 5044
  }
}

output {
  elasticsearch {
    hosts => ["http://192.xxx.xxx.135:9200"]
    index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
    user => "elastic"
    password => "123456"
  }
}
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18

2）测试

./bin/logstash -e 'input { stdin {} } output { stdout {} }' 
1

3）启动服务

nohup /opt/elk/logstash-7.17.1/bin/logstash -f /opt/elk/logstash-7.17.1/config/logstash-sample.conf &

nohup /opt/elk/logstash-7.17.1/bin/logstash >> /opt/elk/logstash-7.17.1/logs/logstash.log &
1
2
3

4）查看logstash运行状态

ln -s /opt/elk/logstash-7.17.1/bin/logstash.lib.sh /bin/
1

3.11.filebeat

1）配置

vim /opt/elk/filebeat-7.17.1/filebeat.yml

1
2

2）设置

ll /etc/sudoers
vim /etc/sudoers

# 添加
admin    ALL=(ALL)       ALL
es    ALL=(ALL)       ALL

chmod u-w /etc/sudoers

sudo filebeat modules enable kibana
1
2
3
4
5
6
7
8
9
10

4.客户端（Client）

4.1.Python

python -m pip install elasticsearch==7.17.1

python -m pip uninstall elasticsearch==8.1.0
1
2
3

1）测试

# 安装雪花算法库
python -m pip install pysnowflake
1
2

附录

A1.参考

• elasticsearch
• kibana
• logstash
• Beats
  • filebeat
  • packetbeat
  • winlogbeat
  • metricbeat
  • heartbeat
  • auditbeat
  • functionbeat
• elasticsearch 自定义JVM参数
• elk安装