ELK 日志收集

方案

1. 部署 FileBeat 从集群每个节点采集日志
2. FileBeat 发送日志到 ElasticSearch 并保存
3. 部署 Kibana 展示 ElasticSearch 数据

 

部署 FileBeat

helm部署

官方文档： https://github.com/elastic/helm-charts/tree/master/filebeat

部署命令

helm repo add elastic https://helm.elastic.co
helm install fb elastic/filebeat -f values.yaml -n kube-system

当前最新为 8.5.1，但本文使用 7.17 版本

values.yaml 文件

配置参考文档：https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-input-container.html

input.paths：模糊匹配需要收集的日志路径
input.fields：定义收集日志的 key
input.fields_under_root：设置 ture，fields存储在输出文档的顶级位置，参考文章 ：filebeat输出结果到elasticsearch的多个索引
setup.template：自定义索引
setup.ilm.enabled：设置 falsel；若启用，则忽略 output.elasticsearch.index 的设置
output.elasticsearch.indices.index：设置 elastic 的索引
output.elasticsearch.indices.when.contains：对应 input 的 fields

filebeatConfig:
  filebeat.yml: |
    filebeat.inputs:
    - type: container
      paths:
        - /var/log/containers/*dev_permission-service*.log
      fields:
        app_id: "dev-permission"
      fields_under_root: true
        
    - type: container
      paths:
        - /var/log/containers/*dev_converter-service*.log
      fields:
        app_id: "dev-converter"
      fields_under_root: true
      
    setup.template.enabled: true
    setup.template.fields: fields.yml
    setup.template.name: "k8s"
    setup.template.pattern: "k8s-*"
    setup.template.enabled: true
    setup.ilm.enabled: false

    output.elasticsearch:
      index: "k8s-other"
      username: 'elastic'
      password: 'xxxxx'
      protocol: http
      hosts: ["http://elsmy.saas.api.gd-njc.com:80"]
      indices:
      - index: "k8s-dev-permission-%{+yyyy.MM.dd}"
        when.contains:
          app_id: "dev-permission"
      - index: "k8s-dev-converter-%{+yyyy.MM.dd}"
        when.contains:
          app_id: "dev-converter"

 

yaml 部署

官方文档：https://www.elastic.co/guide/en/beats/filebeat/current/running-on-kubernetes.html

编辑 filebeat-kubernetes.yaml

---
apiVersion: v1
kind: ConfigMap
metadata:
  name: filebeat-config
  namespace: kube-system
  labels:
    k8s-app: filebeat
data:
  filebeat.yml: |-
    filebeat.inputs:
    - type: container
      paths:
        - /var/log/containers/*dev_permission-service*.log
      fields:
        app_id: "dev-permission"
      fields_under_root: true
        
    - type: container
      paths:
        - /var/log/containers/*dev_converter-service*.log
      fields:
        app_id: "dev-converter"
      fields_under_root: true
      
    setup.template.enabled: true
    setup.template.fields: fields.yml
    setup.template.name: "k8s"
    setup.template.pattern: "k8s-*"
    setup.template.enabled: true
    setup.ilm.enabled: false

    output.elasticsearch:
      index: "k8s-other"
      username: 'elastic'
      password: 'xxxxx'
      protocol: http
      hosts: ["http://elsmy.saas.api.gd-njc.com:80"]
      indices:
      - index: "k8s-dev-permission-%{+yyyy.MM.dd}"
        when.contains:
          app_id: "dev-permission"
      - index: "k8s-dev-converter-%{+yyyy.MM.dd}"
        when.contains:
          app_id: "dev-converter"
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: filebeat
  namespace: kube-system
  labels:
    k8s-app: filebeat
spec:
  selector:
    matchLabels:
      k8s-app: filebeat
  template:
    metadata:
      labels:
        k8s-app: filebeat
    spec:
      serviceAccountName: filebeat
      terminationGracePeriodSeconds: 30
      hostNetwork: true
      dnsPolicy: ClusterFirstWithHostNet
      containers:
      - name: filebeat
        image: docker.elastic.co/beats/filebeat:8.4.2
        args: [
          "-c", "/etc/filebeat.yml",
          "-e",
        ]
        env:
        - name: ELASTICSEARCH_HOST
          value: elasticsearch
        - name: ELASTICSEARCH_PORT
          value: "9200"
        - name: ELASTICSEARCH_USERNAME
          value: elastic
        - name: ELASTICSEARCH_PASSWORD
          value: changeme
        - name: ELASTIC_CLOUD_ID
          value:
        - name: ELASTIC_CLOUD_AUTH
          value:
        - name: NODE_NAME
          valueFrom:
            fieldRef:
              fieldPath: spec.nodeName
        securityContext:
          runAsUser: 0
          # If using Red Hat OpenShift uncomment this:
          #privileged: true
        resources:
          limits:
            memory: 200Mi
          requests:
            cpu: 100m
            memory: 100Mi
        volumeMounts:
        - name: config
          mountPath: /etc/filebeat.yml
          readOnly: true
          subPath: filebeat.yml
        - name: data
          mountPath: /usr/share/filebeat/data
        - name: varlibdockercontainers
          mountPath: /var/lib/docker/containers
          readOnly: true
        - name: varlog
          mountPath: /var/log
          readOnly: true
      volumes:
      - name: config
        configMap:
          defaultMode: 0640
          name: filebeat-config
      - name: varlibdockercontainers
        hostPath:
          path: /var/lib/docker/containers
      - name: varlog
        hostPath:
          path: /var/log
      # data folder stores a registry of read status for all files, so we don't send everything again on a Filebeat pod restart
      - name: data
        hostPath:
          # When filebeat runs as non-root user, this directory needs to be writable by group (g+w).
          path: /var/lib/filebeat-data
          type: DirectoryOrCreate
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: filebeat
subjects:
- kind: ServiceAccount
  name: filebeat
  namespace: kube-system
roleRef:
  kind: ClusterRole
  name: filebeat
  apiGroup: rbac.authorization.k8s.io
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: filebeat
  namespace: kube-system
subjects:
  - kind: ServiceAccount
    name: filebeat
    namespace: kube-system
roleRef:
  kind: Role
  name: filebeat
  apiGroup: rbac.authorization.k8s.io
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: filebeat-kubeadm-config
  namespace: kube-system
subjects:
  - kind: ServiceAccount
    name: filebeat
    namespace: kube-system
roleRef:
  kind: Role
  name: filebeat-kubeadm-config
  apiGroup: rbac.authorization.k8s.io
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: filebeat
  labels:
    k8s-app: filebeat
rules:
- apiGroups: [""] # "" indicates the core API group
  resources:
  - namespaces
  - pods
  - nodes
  verbs:
  - get
  - watch
  - list
- apiGroups: ["apps"]
  resources:
    - replicasets
  verbs: ["get", "list", "watch"]
- apiGroups: ["batch"]
  resources:
    - jobs
  verbs: ["get", "list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: filebeat
  # should be the namespace where filebeat is running
  namespace: kube-system
  labels:
    k8s-app: filebeat
rules:
  - apiGroups:
      - coordination.k8s.io
    resources:
      - leases
    verbs: ["get", "create", "update"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: filebeat-kubeadm-config
  namespace: kube-system
  labels:
    k8s-app: filebeat
rules:
  - apiGroups: [""]
    resources:
      - configmaps
    resourceNames:
      - kubeadm-config
    verbs: ["get"]
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: filebeat
  namespace: kube-system
  labels:
    k8s-app: filebeat
---

部署命令

kubectl apply -f filebeat-kubernetes.yaml

 

部署 Elastic

官网：https://www.elastic.co/guide/en/cloud-on-k8s/current/k8s-quickstart.html

安装 eck

官网：https://www.elastic.co/guide/en/cloud-on-k8s/current/k8s-deploy-eck.html

安装 crds 和 operator

kubectl create -f https://download.elastic.co/downloads/eck/2.5.0/crds.yaml
kubectl apply -f https://download.elastic.co/downloads/eck/2.5.0/operator.yaml

查看 operator 运行日志

kubectl -n elastic-system logs -f statefulset.apps/elastic-operator

 

安装 elastic

部署命令

kubectl apply -f deployment.yaml

deployment.yaml 文件

namespace：配置命名空间
tls：关闭 tls 协议，配置文章：https://www.elastic.co/guide/en/cloud-on-k8s/current/k8s-transport-settings.html
storageClassName：配置存储卷，可以参考文章 nfs-client
storage：配置存储空间，预设 10Gi，配置文章：https://www.elastic.co/guide/en/cloud-on-k8s/current/k8s-volume-claim-templates.html
resources：配置最大最小内存，配置文章：https://www.elastic.co/guide/en/cloud-on-k8s/current/k8s-managing-compute-resources.html
ES_JAVA_OPTS：配置最大最小 java 内存，配置文章：https://www.elastic.co/guide/en/cloud-on-k8s/current/k8s-jvm-heap-dumps.html

apiVersion: elasticsearch.k8s.elastic.co/v1
kind: Elasticsearch # 需要安装es operator
metadata:
  name: es
  namespace: els-test
spec:
  version: 7.11.2
  http:
    tls:
      selfSignedCertificate:
        disabled: true
  nodeSets:
    - name: es
      count: 3
      volumeClaimTemplates:
        - metadata:
            name: elasticsearch-data
          spec:
            accessModes:
              - ReadWriteOnce
            resources:
              requests:
                storage: 10Gi
            storageClassName: nfs-client
      podTemplate:
        spec:           
          containers:
            - name: elasticsearch
              env:
                - name: ES_JAVA_OPTS
                  value: "-Xms2g -Xmx2g"
              resources:
                requests:
                  memory: 3Gi
                limits:
                  memory: 3Gi

部署结果

解决报错

报错信息：

ERROR: [1] bootstrap checks failed
[1]: max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]

解决报错

1. 修改配置： vi /etc/sysctl.conf
2. 添加配置： vm.max_map_count=655360
3. 并=执行命令： sysctl -p
4. 检查：sysctl -a | grep vm.max_map_count

登录
账号：elastic
密码：

 kubectl  get secret es-es-elastic-user -n els-test -o=jsonpath='{.data.elastic}' | base64 --decode 

配置 ingress 或 nodeport ，参考文章：ingress 部署

结果如下即 elastic 部署成功，cluster_name 是 es

查看elastic集群索引信息：/_cat/indices?v

 

部署 Kibana

部署命令

kubectl apply -f kibana.yaml

kibana.yaml

namespace：和 elastic 一样
tls：关闭 tls 协议
elasticsearchRef：和 elastic 的 cluster_name 一样
编写 deployment.yaml 文件

apiVersion: kibana.k8s.elastic.co/v1
kind: Kibana
metadata:
  name: kibana
  namespace: els-test
spec:
  version: 7.11.2
  count: 1
  elasticsearchRef:
    name: es
  http:
    tls:
      selfSignedCertificate:
        disabled: true

部署结果

配置 ingress 或 nodeport ，参考文章：ingress 部署

配置索引正则： [manage space] -> [index patterns]

创建索引正则匹配

在 discover 可以选择配置好的正则

选择 message，只看日志消息