(资料图)

主要记录下filebeat+logstash+elasticsearch+kibana抽取过滤存储展示应用日志文件的方式；版本基于8.8，并开启xpack安全认证。由于从7.X开始就自带JDK，故这里也不展示环境配置等步骤。

elasticsearch：https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-8.8.1-linux-x86_64.tar.gzkibana:https://artifacts.elastic.co/downloads/kibana/kibana-8.8.1-linux-x86_64.tar.gzfilebeat:https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-8.8.1-linux-x86_64.tar.gzlogstash:https://artifacts.elastic.co/downloads/logstash/logstash-8.8.1-linux-x86_64.tar.gz

useradd -d /home/elk -m elkecho "123@qwe"|passwd elk --stdin

- 创建es数据目录和log目录；mkdir /data/elk/elasticsearch/{data,log}- 解压安装包tar -zxvf elasticsearch-8.8.1-linux-x86_64.tar.gz- 进入config目录，修改配置文件elasticsearch.yml；cd elasticsearch-8.8.1/config- 修改配置文件取消以下注释并配置；vim elasticsearch.ymlcluster.name: my-applicationnode.name: node-1path.data: /data/elk/elasticsearch/datapath.logs: /data/elk/elasticsearch/logsnetwork.host: 0.0.0.0http.port: 9200- 首次启动不要后台启动；pwd/home/elk/elasticsearch-8.8.1/configcd /home/elk/elasticsearch-8.8.1/bin./elasticsearch**前台日志输出最后内容<要记录下来>**：✅ Elasticsearch security features have been automatically configured!✅ Authentication is enabled and cluster connections are encrypted.ℹ️  Password for the elastic user (reset with `bin/elasticsearch-reset-password -u elastic`):  2j6qweqeRqnAnPGU61ℹ️  HTTP CA certificate SHA-256 fingerprint:  09189c0bb24353451b32f603d509272d591sad123815b1233d7aeℹ️  Configure Kibana to use this cluster:• Run Kibana and click the configuration link in the terminal when Kibana starts.• Copy the following enrollment token and paste it into Kibana in your browser (valid for the next 30 minutes):  eyJ2ZXIiOiI4LjguMSIsImFkciI6WyIxMC4yNTMuMTc3LjkyOjkyMDAiXSwiZmdyIjoiMDkxODljMGJiMjc4NDE4YTIyNjE4YjBlN2M5OGIzMmY2MDNkNTA5MjcyZDU5MWZiNzkwMDQzODE1YjY3ZDdhZSIsImtleSI6Im02ckE5WWdCUEJtZ2J3czVUWU14OjRUYVliMi1SUWFHSlVlRWJaYk5NUVEifQ==ℹ️ Configure other nodes to join this cluster:• Copy the following enrollment token and start new Elasticsearch nodes with `bin/elasticsearch --enrollment-token ` (valid for the next 30 minutes):  eyJ2ZXIiOiI4LjguMSIsImFkciI6WyIxMC4yNTMuMTc3LjkyOjkyMDAiXSwiZmdyIjoiMDkxODljMGJiMjc4NDE4YTIyNjE4YjBlN2M5OGIzMmY2MDNkNTA5MjcyZDU5MWZiNzkwMDQzODE1YjY3ZDdhZSIsImtleSI6Im1xckE5WWdCUEJtZ2J3czVUWU12Omt1aEdkVXAzUTA2LUpqOVNmMWkweEEifQ==  If you"re running in Docker, copy the enrollment token and run:  `docker run -e "ENROLLMENT_TOKEN=" docker.elastic.co/elasticsearch/elasticsearch:8.8.1`- 重新开启一个会话窗口，再次查看elasticsearch.yml配置，会发现多了xpack安全认证；    ```    # Enable security features    xpack.security.enabled: true        xpack.security.enrollment.enabled: true        xpack.monitoring.collection.enabled: true        # Enable encryption for HTTP API client connections, such as Kibana, Logstash, and Agents    xpack.security.http.ssl:      enabled: true      keystore.path: certs/http.p12        # Enable encryption and mutual authentication between cluster nodes    xpack.security.transport.ssl:      enabled: true      verification_mode: certificate      keystore.path: certs/transport.p12      truststore.path: certs/transport.p12    # Create a new cluster with the current node only    # Additional nodes can still join the cluster later    cluster.initial_master_nodes: ["node-1"]    ```    同时/home/elk/elasticsearch-8.8.1/config目录下增加一个certs目录，里面有以下内容；    ll certs/    总用量 24    -rw-rw---- 1 elk elk 1915 6月  26 11:29 http_ca.crt    -rw-rw---- 1 elk elk 9997 6月  26 11:29 http.p12    -rw-rw---- 1 elk elk 5822 6月  26 11:29 transport.p12    - 关闭elasticsearch服务，并后台启动；[elk@host-172-16-0-1 config]$ ps -ef|grep elasticsearch|grep -v grep|awk "{print $2}"|xargs kill[elk@host-172-16-0-1 config]$ cd  ../bin/[elk@host-172-16-0-1 bin]$ ./elasticsearch -d - 初始化elasticsearch内置kibana用户密码；./elasticsearch-reset-password -u kibaina- 浏览器访问https://172.16.0.1:9200，输入elastic密码2j6qweqeRqnAnPGU61：

- 解压安装包；tar -zxvf kibana-8.8.1-linux-x86_64.tar.gzcd  kibana-8.8.1/config- 拷贝es目录下certs文件夹到config下；\cp  -rf /home/elk/elasticsearch-8.8.1/config/certs ./- 修改kibana.yml，取消以下注释并配置；vim kibana.ymlserver.port: 5601server.host: "172.16.0.1"elasticsearch.hosts: ["https://172.16.0.1:9200"]elasticsearch.username: "kibana"    # es内置用户；elasticsearch.password: "pkRqnAnPGU61123"  # es初始化的的密码；elasticsearch.ssl.certificateAuthorities: [ "/home/elk/kibana-8.8.1/config/certs/http_ca.crt" ]i18n.locale: "zh-CN"- 启动kibanacd  /home/elk/kibana-8.8.1/nohup ./bin/kibana &- 访问http://10.253.177.92:5601/login，输入elastic密码2j6qweqeRqnAnPGU61：

- 解压安装包；tar -zxvf logstash-8.8.1-linux-x86_64.tar.gzcd logstash-8.8.1/config/- 拷贝es目录下certs文件夹到config下；\cp  -rf /home/elk/elasticsearch-8.8.1/config/certs ./- 修改配置文件logstash.yml；vim logstash.ymlhttp.host: "0.0.0.0"xpack.monitoring.enabled: truexpack.monitoring.elasticsearch.username: elasticxpack.monitoring.elasticsearch.password: 2j6qweqeRqnAnPGU61xpack.monitoring.elasticsearch.hosts: ["https://172.16.0.1:9200"]xpack.monitoring.elasticsearch.ssl.certificate_authority: "/home/elk/logstash-8.8.1/config/certs/http_ca.crt"xpack.monitoring.elasticsearch.ssl.ca_trusted_fingerprint: 09189c0bb278418a22618b0e7c98b32f603d509272d591fb790043815b67d7ae - 修改logstash-sample.conf配置；vim logstash-sample.confinput {  beats {    port => 5041  }}output {  elasticsearch {    hosts => ["https://172.16.0.1:9200"]    #index => "%{[fields][service_name]}-%{+YYYY.MM.dd}"    ssl => true    ssl_certificate_verification => false    cacert => "/home/elk/logstash-8.8.1/config/certs/http_ca.crt"    ca_trusted_fingerprint => "09189c0bb278418a22618b0e7c98b32f603d509272d591fb790043815b67d7ae"    user => "elastic"    password => "2j6qweqeRqnAnPGU61"  }  stdout {codec => rubydebug}}- 启动logstash；nohup ./bin/logstash -f /home/elk/logstash-8.8.1/config/logstash-sample.conf &

- 分别登录172.16.0.2/3两台应用服务器/data目录；mkdir filebeattar -zxvf filebeat-8.8.1-linux-x86_64.tar.gzcd filebeat-8.8.1-linux-x86_64- 修改filebeat.yml配置文件vim filebeat.ymlfilebeat.inputs: - type: log id: 1 enable: true paths:    - /data/app/ap/logs/*.log  # 要采集的日志文件或路径# output.elasticsearch:  # 由于本文架构是filebeat的output是到logstash，故关闭默认output.elasticsearch；output.logstash:    hosts: ["172.16.0.1:5041"]  # 这里的端口要与logstash-sample.conf配置里的一致；- 配置完成，临时启动filebeat；nohup ./filebeat -e -c filebeat.yml > /dev/null 2>&1- 由于通过nohub方式启动filebeat，运行一段时间后filebeat自动退出；原因是filebeat默认会定期检测文件是否有新的内容，如果超过一定时间检测的文件没有新数据写入，那么filebeat会自动退出，解决办法就是将filebeat通过系统后台的方式长期运行；    - 添加systemctl服务启动配置    vim  /etc/systemd/system/filebeat.service        [Unit]    Description=Filebeat is a lightweight shipper for metrics.    Documentation=https://www.elastic.co/products/beats/filebeat    Wants=network-online.target    After=network-online.target        [Service]    Environment="LOG_OPTS=-e"    Environment="CONFIG_OPTS=-c /data/filebeat/filebeat-8.8.1-linux-x86_64/filebeat.yml"    Environment="PATH_OPTS=-path.home /data/filebeat/filebeat-8.8.1-linux-x86_64/filebeat -path.config /data/filebeat/fileb    eat-8.8.1-linux-x86_64 -path.data /data/filebeat/filebeat-8.8.1-linux-x86_64/data -path.logs /data/filebeat/filebeat-8.    8.1-linux-x86_64/logs"    ExecStart=/data/filebeat/filebeat-8.8.1-linux-x86_64/filebeat $LOG_OPTS $CONFIG_OPTS $PATH_OPTS    Restart=always        [Install]    WantedBy=multi-user.target        - 授予可执行权限    chmod +x /etc/systemd/system/filebeat.service        - 配置开机启动等    systemctl daemon-reload    systemctl enable filebeat    systemctl start filebeat

关键词：