Asp-Net-Core-搭建ELK日志平台-Docker-Compose版本

title: Asp.Net Core 搭建ELK日志平台(Docker-Compose版本)

date: 2022-09-27 15:16:59

tags:

- .NET

由于暂时用不上Logstash的过滤功能，因此本部署方案直接使用Nlog输出到ElasticSearch，项目的架构如下：

docker-compose部署

部署文件内容如下：

version: "3" # 不同版本 支持 docker 版本也不同

services:

  elasticsearch:

    image: elasticsearch:7.5.0 # 镜像

    container_name: "elasticsearch" # 容器名称

    ports: # 绑定端口

      -  9200:9200

      -  9300:9300

    environment: # 环境参数

      - "discovery.type=single-node" # 单节点 而不是集群

    volumes: # 挂载卷

      - F:/elasticsearch/data:/usr/share/elasticsearch/data

      - F:/elasticsearch/plugins:/usr/share/elasticsearch/plugins

  elastalert:

    container_name: "elastalert"

    image: karql/elastalert2-server:latest

    ports:

      - 3030:3030

      - 3333:3333

    links: # 链接容器,即在该容器内的hosts文件添加一条 xxx.xxx.xxx.xxx elasticsearch 的dns地址

      - elasticsearch

    depends_on: # 容器依赖，使得elastalert在elasticsearch后部署启动

      - elasticsearch

    volumes:

      - F:/elasticsearch/elastalert/config/elastalert.yaml:/opt/elastalert/config.yaml

      - F:/elasticsearch/elastalert/config/elastalert-test.yaml:/opt/elastalert/config-test.yaml

      - F:/elasticsearch/elastalert/config/config.json:/opt/elastalert-server/config/config.json

      - F:/elasticsearch/elastalert/rules:/opt/elastalert/rules

      - F:/elasticsearch/elastalert/rule_templates:/opt/elastalert/rule_templates

  kibana:

    container_name: "kibana"

    image: kibana:7.5.0

    ports:

      - 5601:5601

    links:

      - elasticsearch

      - elastalert

    depends_on:

      - elasticsearch

      - elastalert

    volumes:

      - F:\ELKDeploy\elasticsearch\kibana.yml:/usr/share/kibana/config/kibana.yml

      - F:\ELKDeploy\elasticsearch\kibana-plugins\plugins:/usr/share/kibana/plugins

记得替换掉挂载卷的路径，以上配置文件我都放在了我的github上：https://github.com/li-zheng-hao/ELKOnDocker.git ，直接clone下来就行，所有的配置文件我都打包成了zip，记得解压缩

如果在linux下有可能还需要配置对应的文件夹访问权限 chmod -R 777 ./ELKOnDocker

然后在该文件目录下运行命令：

docker compose -f deploy.yml up  -d

等待安装完成即可

Webapi项目配置

添加依赖库

NLog.Web.AspNetCore
NLog.Targets.ElasticSearch

配置日志config文件

新建nlog.config文件:

<nlog xmlns="http://www.nlog-project.org/schemas/NLog.xsd" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">

	  <extensions>

		  <add assembly="NLog.Targets.ElasticSearch"/>

	  </extensions>

    <targets>

      <target xsi:type="File" name="f" fileName="${basedir}/App_Data/logs/${shortdate}.log" layout="${date:format=HH\:mm\:ss}|${level}|${message}" />

		<target name="elastic" xsi:type="BufferingWrapper" flushTimeout="5000">

			<target xsi:type="ElasticSearch" index="webapi-${date:format=yyyy.MM.dd}" includeAllProperties="true" uri="http://172.10.2.111:9200/" >

				<field name="MachineName" layout="${machinename}" />

			</target>

		</target>

    </targets>

    <rules>

		<logger name="*" minlevel="Debug" writeTo="elastic" />

      <logger name="*" minlevel="Debug" writeTo="f" />

      <logger name="*" minlevel="Debug" writeTo="cc" />

    </rules>

  </nlog><?xml version="1.0" encoding="utf-8" ?>

<nlog xmlns="http://www.nlog-project.org/schemas/NLog.xsd"

      xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"

      internalLogToConsole="true">

	<extensions>

		<add assembly="NLog.Targets.ElasticSearch"/>

	</extensions>

	<targets>

		 <!--uri="http://localhost:9200"-->

		<!--ElasticSearch保存日志信息-->

        <target name="elastic" xsi:type="BufferingWrapper" flushTimeout="5000">

            <target name="elk" xsi:type="ElasticSearch"

                    ConnectionStringName="ElasticSearchServerAddress"

                    index="webapi-${shortdate}"

                    layout ="API:MyServiceName|${longdate}|${event-properties:item=EventId_Id}|${uppercase:${level}}|${logger}|${message} ${exception:format=tostring}|url: ${aspnet-request-url}|action: ${aspnet-mvc-action}">

                <field name="MachineName" layout="${machinename}" />

                <field name="Time" layout="${longdate}" />

                <field name="level" layout="${level:uppercase=true}" />

                <field name="logger" layout=" ${logger}" />

                <field name="message" layout=" ${message}" />

            </target>

        </target>

	</targets>

	<rules>

		<logger name="Microsoft.*" maxLevel="Info" final="true" />

		<logger name="*" minLevel="Debug" writeTo="elastic" ></logger>

	</rules>

</nlog>

在appsettings.json内配置对应的地址:

{

  "Logging": {

    "LogLevel": {

      "Default": "Information",

      "Microsoft": "Warning",

      "Microsoft.Hosting.Lifetime": "Information"

    }

  },

  "AllowedHosts": "*",

  "ConnectionStrings": {

    "ElasticSearchServerAddress": "http://localhost:9200" //新增 根据自己搭建的地址替换

  }

}

添加Nlog替换自带的logger

public class Program

{

    public static void Main(string[] args)

    {

        NLogBuilder.ConfigureNLog("nlog.config");// 新增

        CreateHostBuilder(args).Build().Run();

    }

    public static IHostBuilder CreateHostBuilder(string[] args) =>

        Host.CreateDefaultBuilder(args)

        .ConfigureWebHostDefaults(webBuilder =>

                                  {

                                      webBuilder.UseStartup<Startup>().UseNLog();//新增

                                  });

}

测试

[ApiController]

[Route("[controller]")]

public class WeatherForecastController : ControllerBase

{

    private static readonly string[] Summaries = new[]

    {

        "Freezing", "Bracing", "Chilly", "Cool", "Mild", "Warm", "Balmy", "Hot", "Sweltering", "Scorching"

    };

    private readonly ILogger<WeatherForecastController> _logger;

    public WeatherForecastController(ILogger<WeatherForecastController> logger)

    {

        _logger = logger;

    }

    [HttpGet]

    public string Get(string data)

    {

        _logger.LogDebug("写入测试数据");

        return "ok";

    }

}

调用接口写入数据，打开kibana查看：

数据成功写入

配置日志异常报警

如果是用的我的compose文件及配置文件的话，部署完成后插件应该是直接安装完成了的，进入kibana页面后，如下图显示，如果有elastalert说明插件安装成功：

配置http请求报警

在kibana的elastalert页面内加入如下规则：

name: cutsom_rule

type: any

index: webapi-*

num_events: 1

timeframe:

  minutes: 1

filter:

- query:

    query_string:

      query: "message: 错误  OR　error OR ERROR"

alert: post

http_post_url: "http://host.docker.internal:5000/WeatherForecast"

http_post_payload:

    content: message

http_post_static_payload:

  test_str: "my test string"

这里说明一下上述的配置字段，index为日志的索引，query也就是满足KQL查询的规则的字符串，http_pot_url填入异常通知调用的接口，由于我的webapi服务没有部署在docker内，这里需要通过host.docker.internal来自动获取容器宿主机的ip

更多的参数说明参考elastalert的官方手册：https://elastalert.readthedocs.io/en/latest/ruletypes.html?highlight=http#http-post

配置完成后会自动监测日志

巴特西