Filebeat Processors对日志数据应用基本处理和数据增强功能

下面是一个使用drop_fields处理器从Apache访问日志中删除一些字段的示例：

filebeat.inputs:

- type: log

  enabled: true

  fields:

    apache: true

  tags: ["my-service", "hardware", "test"]

  paths:

    - /Users/liuxg/data/apache-daily-access.log

processors: # 注意这几行

  - drop_fields:

      fields: ["ecs"]

output.elasticsearch:

  hosts: ["localhost:9200"]

在上面，把ecs字段删除，那么显示的结果为：

{

        "_index" : "filebeat-7.3.0-2019.09.11-000001",

        "_type" : "_doc",

        "_id" : "m4H8IG0BJD_DqHjgZ47a",

        "_score" : 1.0,

        "_source" : {

          "@timestamp" : "2019-09-11T15:41:15.306Z",

          "host" : {

            "name" : "localhost"

          },

          "agent" : {

            "type" : "filebeat",

            "ephemeral_id" : "d32d0cea-966a-48d7-8728-dad5fc276b3a",

            "hostname" : "localhost",

            "id" : "c88813ba-fdea-4a98-a0be-468fb53566f3",

            "version" : "7.3.0"

          },

          "log" : {

            "offset" : 11497,

            "file" : {

              "path" : "/Users/liuxg/data/apache-daily-access.log"

            }

          },

          "message" : """164.51.31.185 - - [11/Sep/2019:00:04:15 +0000] "GET /item/giftcards/232 HTTP/1.1" 200 130 "/category/electronics" "Mozilla/5.0 (Windows NT 6.0) AppleWebKit/535.11 (KHTML, like Gecko) Chrome/17.0.963.56 Safari/535.11"""",

          "tags" : [

            "my-service",

            "hardware",

            "test"

          ],

          "input" : {

            "type" : "log"

          },

          "fields" : {

            "apache" : true

          }

        }

      }

显然相比较之前的source，我们可以看出来ecs项已经不见了。

所有的Processors在一下列出：

add_cloud_metadata
add_locale
decode_json_fields
drop_event
drop_fields
include_fields
add_kubernetes_metadata
add_docker_metadata

巴特西

Filebeat Processors对日志数据应用基本处理和数据增强功能

最新文章

热门文章