服务端安装

Elasticsearch和Kibana（需要安装openjdk1.8以上）

安装方法：https://www.elastic.co
以Ubuntu为例：

wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -

sudo apt-get install apt-transport-https

echo "deb https://artifacts.elastic.co/packages/6.x/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-6.x.list

apt-get update

apt-get install elasticsearch

apt-get install kibana

elasticsearch配置

cat /etc/elasticsearch/elasticsearch.yml

path.data: /var/lib/elasticsearch

path.logs: /var/log/elasticsearch

network.host: 172.16.80.28

http.port: 9200

discovery.zen.ping.unicast.hosts: ["172.16.80.28","172.16.80.155"]

附件有文件elasticsearch.map

Kibana配置

cat /etc/kibana/kibana.yml

server.port: 5601

server.host: "172.16.80.28"

elasticsearch.url: "http://172.16.80.28:9200"

Elastalert（需要Python2.7）

下载 https://github.com/Yelp/elastalert/releases

进入elastalert目录

pip install -r requirements.txt

python setup.py install

elastalert-create-index

前台运行elastalert

python -m elastalert.elastalert --verbose --rule rules/my_rule.yaml

安装参考:
https://elastalert.readthedocs.io/en/latest/running_elastalert.html#downloading-and-configuring
https://blog.csdn.net/df007df/article/details/54773391

elastalert配置

cat /etc/elastalert/rules/my_rule.yaml

es_host: 172.16.80.28

es_port: 9200

name: filebeat rules

type: frequency

index: filebeat*

num_events: 5

timeframe:

  hours: 1

filter:

- query:

    query_string:

      query: "message: *error*"

alert:

- "email"

email:

- "aaa@qq.com"

- "bbb@qq.com"

alert_text: "Ref Log http://172.16.80.28:5601/app/kibana"

smtp_host: smtp.exmail.qq.com

smtp_port: 25

smtp_ssl: false

smtp_auth_file: /etc/elastalert/rules/smtp_auth_file.yaml

from_addr:  aaa@qq.com

cat /etc/elastalert/rules/smtp_auth_file.yaml

user: "aaa@qq.com"

password: "邮箱密码"

cat /etc/elastalert/config.yaml

rules_folder: /etc/elastalert/rules

run_every:

  minutes: 1

buffer_time:

  minutes: 15

es_host: 172.16.80.28

es_port: 9200

s_url_prefix: elasticsearch

writeback_index: elastalert_status

alert_time_limit:

  days: 2

elastalert配置注解

rules_folder：用来加载下一阶段rule的设置，默认是example_rules

run_every：用来设置定时向elasticsearch发送请求

buffer_time：用来设置请求里时间字段的范围，默认是45分钟

es_host：elasticsearch的host地址

es_port：elasticsearch 对应的端口号

use_ssl：可选的，选择是否用SSL连接es，true或者false

es_username：es认证的username

es_password：es认证的password

writeback_index：elastalert产生的日志在elasticsearch中的创建的索引

alert_time_limit：失败重试的时间限制

告警参考
https://blog.csdn.net/gamer_gyt/article/details/52917116
https://elastalert.readthedocs.io/en/latest/ruletypes.html

客户端安装

Filebeat安装

以Ubuntu为例：

wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -

sudo apt-get install apt-transport-https

echo "deb https://artifacts.elastic.co/packages/6.x/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-6.x.list

apt-get update

apt-get install filebeat

filebeat配置

cat /etc/filebeat/filebeat.yml

filebeat.prospectors:

- type: log

  enabled: true

  paths:

    - /var/log/nginx/*.log

  path: ${path.config}/modules.d/*.yml

  reload.enabled: false

setup.template.settings:

  index.number_of_shards: 3

setup.kibana:

  host: "172.16.80.28:5601"

output.elasticsearch:

  hosts: ["172.16.80.28:9200"]

EFK备注

日志位置 /var/lib/elasticsearch/nodes/
日志大小暂时为345Mb
日志轮循暂时不知道在哪设置

elasticsearch.map内容：

# uint mapping

{

   "settings" : {

      "index" : {

         "number_of_replicas" : ,

         "number_of_shards" :

      }

   },

   "mappings" : {

      "values" : {

         "properties" : {

            "itemid" : {

               "type" : "long"

            },

            "clock" : {

               "format" : "epoch_second",

               "type" : "date"

            },

            "value" : {

               "type" : "long"

            }

         }

      }

   }

}

# dbl mapping

{

   "settings" : {

      "index" : {

         "number_of_replicas" : ,

         "number_of_shards" :

      }

   },

   "mappings" : {

      "values" : {

         "properties" : {

            "itemid" : {

               "type" : "long"

            },

            "clock" : {

               "format" : "epoch_second",

               "type" : "date"

            },

            "value" : {

               "type" : "double"

            }

         }

      }

   }

}

# str mapping

{

   "settings" : {

      "index" : {

         "number_of_replicas" : ,

         "number_of_shards" :

      }

   },

   "mappings" : {

      "values" : {

         "properties" : {

            "itemid" : {

               "type" : "long"

            },

            "clock" : {

               "format" : "epoch_second",

               "type" : "date"

            },

            "value" : {

               "fields" : {

                  "analyzed" : {

                     "index" : true,

                     "type" : "text",

                     "analyzer" : "standard"

                  }

               },

               "index" : false,

               "type" : "text"

            }

         }

      }

   }

}

# text mapping

{

   "settings" : {

      "index" : {

         "number_of_replicas" : ,

         "number_of_shards" :

      }

   },

   "mappings" : {

      "values" : {

         "properties" : {

            "itemid" : {

               "type" : "long"

            },

            "clock" : {

               "format" : "epoch_second",

               "type" : "date"

            },

            "value" : {

               "fields" : {

                  "analyzed" : {

                     "index" : true,

                     "type" : "text",

                     "analyzer" : "standard"

                  }

               },

               "index" : false,

               "type" : "text"

            }

         }

      }

   }

}

# log mapping

{

   "settings" : {

      "index" : {

         "number_of_replicas" : ,

         "number_of_shards" :

      }

   },

   "mappings" : {

      "values" : {

         "properties" : {

            "itemid" : {

               "type" : "long"

            },

            "clock" : {

               "format" : "epoch_second",

               "type" : "date"

            },

            "value" : {

               "fields" : {

                  "analyzed" : {

                     "index" : true,

                     "type" : "text",

                     "analyzer" : "standard"

                  }

               },

               "index" : false,

               "type" : "text"

            }

         }

      }

   }

}

巴特西

Elasticsearch,Filebeat,Kibana部署,添加图表及elastalert报警