今天我们来聊一下利用docker部署elk日志分析系统，这里解析一下elk是啥东西。elk分别是Elasticsearch,Logstash和Kibana的首字母缩写。
Elasticsearch是一个基于JSON的分布式搜索和分析引擎，专为水平可扩展性，最高可靠性和易管理性而设计。
Logstash是一个动态数据收集管道，具有可扩展的插件生态系统和强大的Elasticsearch协同作用。
Kibana通过UI 提供数据可视化。
架构简述
日志系统首先面临几个问题：
不同厂商设备的不同日志格式的处理，如何调用微信来发送报警信息。采用的解决办法是不同厂商的设备发送日志的时候采用不同端口，日志先发送到logstash,
logstash会先解析日志成标准格式，然后logstash会做2件事情，一个是存放日志到es里面，通过kibana做出展示。
环境搭建：
我们这里采用docker容器技术来部署elk。先从docker hub 下载 Elasticsearch,Logstash和Kibana这三个镜像。执行如下命令
[root@node1 ~]# docker pull elasticsearch[root@node1 ~]# docker pull kibana[root@node1 ~]# docker pull logstash:7.2.0
为了后续管理方便，这里编写yml文件，然后使用docker-compose来启动。
version: '3.3'services: elasticsearch: image: elasticsearch:latest container_name: elasticsearch hostname: elasticsearch ports: - '9200:9200' environment: ES_JAVA_OPTS: "-Xms256m -Xmx256m" discovery.type: "single-node" volumes: - "/data/elk/elasticsearch/data:/usr/elasticsearch/data" logstash: image: logstash:7.2.0 container_name: logstash hostname: logstash ports: - "514:514/udp" user: 'root' command: "logstash -f /etc/logstash.conf --config.reload.automatic" volumes: - "/data/elk/logstash/logstash.conf:/etc/logstash.conf" environment: LS_JAVA_OPTS: "-Xmx256m -Xms256m" links: - "elasticsearch:elasticsearch" kibana: image: kibana:latest container_name: kibana hostname: kibana ports: - '5601:5601' environment: ELASTICSEARCH_URL: "http://elasticsearch:9200" links: - "elasticsearch:elasticsearch"
注意：/data/elk/logstash/logstash.conf这个文件需要我们提前准备，后面会提到。
交换机配置
我们需要把交换机的日志指定到elk服务器上。
cisco:logging host 10.100.18.18 transport udp port 5002H3C info-center enableinfo-center source default channel 2 trap state off // 必要，不然日志会出现 不符合级别的 alert 日志info-center loghost 10.100.18.18 port 5003语言方法
3078P47b99OZ57
b1H84
抖音外卖41232005/04/10 06:38:11
huaweiinfo-center enableinfo-center loghost 10.100.18.18info-center timestamp log short-dateinfo-center timestamp trap short-date
Logstash 的配置
不同厂商的日志 gork我都写好了，复制过去就能用
input{ tcp {port => 5002 type => "Cisco"} udp {port => 514 type => "HUAWEI"} udp {port => 5002 type => "Cisco"} udp {port => 5003 type => "H3C"}}filter { if [type] == "Cisco"{ grok{ match => { "message" => "<%{BASE10NUM:syslog_pri}>%{NUMBER:log_sequence}: .%{SYSLOGTIMESTAMP:timestamp}: %%{DATA:facility}-%{POSINT:severity}-%{CISCO_REASON:mnemonic}: %{GREEDYDATA:message}" } match => { "message" => "<%{BASE10NUM:syslog_pri}>%{NUMBER:log_sequence}: %{SYSLOGTIMESTAMP:timestamp}: %%{DATA:facility}-%{POSINT:severity}-%{CISCO_REASON:mnemonic}: %{GREEDYDATA:message}" } add_field => {"severity_code" => "%{severity}"} overwrite => ["message"] } } else if [type] == "H3C"{ grok { match => { "message" => "<%{BASE10NUM:syslog_pri}>%{SYSLOGTIMESTAMP:timestamp} %{YEAR:year} %{DATA:hostname} %%%{DATA:vvmodule}/%{POSINT:severity}/%{DATA:digest}: %{GREEDYDATA:message}" } remove_field => [ "year" ] add_field => {"severity_code" => "%{severity}"} overwrite => ["message"] }} else if [type] == "HUAWEI"{ grok { match => { "message" => "<%{BASE10NUM:syslog_pri}>%{SYSLOGTIMESTAMP:timestamp} %{DATA:hostname} %%%{DATA:ddModuleName}/%{POSINT:severity}/%{DATA:Brief}:%{GREEDYDATA:message}"} match => { "message" => "<%{BASE10NUM:syslog_pri}>%{SYSLOGTIMESTAMP:timestamp} %{DATA:hostname} %{DATA:ddModuleName}/%{POSINT:severity}/%{DATA:Brief}:%{GREEDYDATA:message}"} remove_field => [ "timestamp" ] add_field => {"severity_code" => "%{severity}"} overwrite => ["message"] }}mutate { gsub => [ "severity", "0", "Emergency", "severity", "1", "Alert", "severity", "2", "Critical", "severity", "3", "Error", "severity", "4", "Warning", "severity", "5", "Notice", "severity", "6", "Informational", "severity", "7", "Debug" ] }}output{ elasticsearch { index => "syslog-%{+YYYY.MM.dd}" hosts => ["your_ipaddress:9200"] }}
本站仅提供存储服务，所有内容均由用户发布，如发现有害或侵权内容，请
点击举报。
