Kubernetes 中审计策略--事件处理
一、审计日志的策略
1、日志记录阶段
kube-apiserver 是负责接收及相应用户请求的一个组件,每一个请求都会有几个阶段,每个阶段都有对应的日志,当前支持的阶段有:
- RequestReceived - apiserver 在接收到请求后且在将该请求下发之前会生成对应的审计日志。
- ResponseStarted - 在响应 header 发送后并在响应 body 发送前生成日志。这个阶段仅为长时间运行的请求生成(例如 watch)。
- ResponseComplete - 当响应 body 发送完并且不再发送数据。
- Panic - 当有 panic 发生时生成。
也就是说对 apiserver 的每一个请求理论上会有三个阶段的审计日志生成。
2、日志记录级别
当前支持的日志记录级别有:
- None - 不记录日志。
- Metadata - 只记录 Request 的一些 metadata (例如 user, timestamp, resource, verb 等),但不记录 Request 或 Response 的body。
- Request - 记录 Request 的 metadata 和 body。
- RequestResponse - 最全记录方式,会记录所有的 metadata、Request 和 Response 的 body。
3、日志记录策略
在记录日志的时候尽量只记录所需要的信息,不需要的日志尽可能不记录,避免造成系统资源的浪费。
- 一个请求不要重复记录,每个请求有三个阶段,只记录其中需要的阶段
- 不要记录所有的资源,不要记录一个资源的所有子资源
- 系统的请求不需要记录,kubelet、kube-proxy、kube-scheduler、kube-controller-manager 等对 kube-apiserver 的请求不需要记录
- 对一些认证信息(secerts、configmaps、token 等)的 body 不记录
审计记录最初产生于 kube-apiserver 内部。每个请求在不同执行阶段都会生成审计事件;这些审计事件会根据特定策略 被预处理并写入后端。策略确定要记录的内容和用来存储记录的后端。 当
你可以使用 --audit-policy-file
标志将包含策略的文件传递给 kube-apiserver
。 如果不设置该标志,则不记录事件。 注意 rules
字段 必须 在审计策略文件中提供。没有(0)规则的策略将被视为非法配置。
k8s 审计日志的一个示例:
{
"kind": "EventList",
"apiVersion": "audit.k8s.io/v1beta1",
"Items": [
{
"Level": "Request",
"AuditID": "793e7ae2-5ca7-4ad3-a632-19708d2f8265",
"Stage": "RequestReceived",
"RequestURI": "/api/v1/namespaces/default/pods/test-pre-sf-de7cc-0",
"Verb": "get",
"User": {
"Username": "system:unsecured",
"UID": "",
"Groups": [
"system:masters",
"system:authenticated"
],
"Extra": null
},
"ImpersonatedUser": null,
"SourceIPs": [
"192.168.1.11"
],
"UserAgent": "kube-scheduler/v1.12.2 (linux/amd64) kubernetes/73f3294/scheduler",
"ObjectRef": {
"Resource": "pods",
"Namespace": "default",
"Name": "test-pre-sf-de7cc-0",
"UID": "",
"APIGroup": "",
"APIVersion": "v1",
"ResourceVersion": "",
"Subresource": ""
},
"ResponseStatus": null,
"RequestObject": null,
"ResponseObject": null,
"RequestReceivedTimestamp": "2019-01-11T06:51:43.528703Z",
"StageTimestamp": "2019-01-11T06:51:43.528703Z",
"Annotations": null
}
]
}
二、启用审计日志
当前的审计日志支持两种收集方式:保存为日志文件和调用自定义的 webhook,在 v1.13 中还支持动态的 webhook。
1、将审计日志以 json 格式保存到本地文件
apiserver 配置文件的 KUBE_API_ARGS 中需要添加如下参数:
--audit-policy-file=/etc/kubernetes/audit-policy.yaml --audit-log-path=/var/log/kube-audit --audit-log-format=json
日志保存到本地后再通过 fluentd 等其他组件进行收集。
还有其他几个选项可以指定保留审计日志文件的最大天数、文件的最大数量、文件的大小等。
2、将审计日志打到后端指定的 webhook
--audit-policy-file=/etc/kubernetes/audit-policy.yaml --audit-webhook-config-file=/etc/kubernetes/audit-webhook-kubeconfig
webhook 配置文件实际上是一个 kubeconfig,apiserver 会将审计日志发送 到指定的 webhook 后,webhook 接收到日志后可以再分发到 kafka 或其他组件进行收集。
audit-webhook-kubeconfig
示例:
apiVersion: v1
clusters:
- cluster:
server: http://127.0.0.1:8081/audit/webhook
name: metric
contexts:
- context:
cluster: metric
user: ""
name: default-context
current-context: default-context
kind: Config
preferences: {}
users: []
前面提到过,apiserver 的每一个请求会记录三个阶段的审计日志,但是在实际中并不是需要所有的审计日志,官方也说明了启用审计日志会增加 apiserver 对内存的使用量。
audit-policy.yaml
配置示例:
apiVersion: audit.k8s.io/v1 # This is required.
kind: Policy
# Don't generate audit events for all requests in RequestReceived stage.
omitStages:
- "RequestReceived"
rules:
# Log pod changes at RequestResponse level
- level: RequestResponse
resources:
- group: ""
# Resource "pods" doesn't match requests to any subresource of pods,
# which is consistent with the RBAC policy.
resources: ["pods"]
# Log "pods/log", "pods/status" at Metadata level
- level: Metadata
resources:
- group: ""
resources: ["pods/log", "pods/status"] # Don't log requests to a configmap called "controller-leader"
- level: None
resources:
- group: ""
resources: ["configmaps"]
resourceNames: ["controller-leader"] # Don't log watch requests by the "system:kube-proxy" on endpoints or services
- level: None
users: ["system:kube-proxy"]
verbs: ["watch"]
resources:
- group: "" # core API group
resources: ["endpoints", "services"] # Don't log authenticated requests to certain non-resource URL paths.
- level: None
userGroups: ["system:authenticated"]
nonResourceURLs:
- "/api*" # Wildcard matching.
- "/version" # Log the request body of configmap changes in kube-system.
- level: Request
resources:
- group: "" # core API group
resources: ["configmaps"]
# This rule only applies to resources in the "kube-system" namespace.
# The empty string "" can be used to select non-namespaced resources.
namespaces: ["kube-system"] # Log configmap and secret changes in all other namespaces at the Metadata level.
- level: Metadata
resources:
- group: "" # core API group
resources: ["secrets", "configmaps"] # Log all other resources in core and extensions at the Request level.
- level: Request
resources:
- group: "" # core API group
- group: "extensions" # Version of group should NOT be included. # A catch-all rule to log all other requests at the Metadata level.
- level: Metadata
# Long-running requests like watches that fall under this rule will not
# generate an audit event in RequestReceived.
omitStages:
- "RequestReceived"
你可以使用最低限度的审计策略文件在 Metadata
级别记录所有请求:
# 在 Metadata 级别为所有请求生成日志
apiVersion: audit.k8s.io/v1beta1
kind: Policy
rules:
- level: Metadata
kubernetes 启用了审计策略
我们可以使用 Webhooks 将审核日志发送到文件或远程 Web API。
在本文中,我们将强制 kube api-server 将审核日志发送到文件。
K8sMeetup
在 Kubernetes 中启用审计策略(对于审计日志文件)
- 创建审计策略 YAML 文件:前往 Kubernetes 集群,并使用以下规则创建 audit-policy.yaml:
apiVersion: audit.k8s.io/v1
kind: Policy
rules:
# Log the request body of configmap changes in kube-system.
- level: Request
resources:
- group: "" # core API group
resources: ["configmaps"]
namespaces: ["kube-system"] # Log configmap and secret changes in all other namespaces at the Metadata level.
- level: Metadata
resources:
- group: "" # core API group
resources: ["secrets", "configmaps"] # A catch-all rule to log all other requests at the Metadata level.
- level: Metadata
omitStages:
- "RequestReceived"
更新 kube api-server 清单文件:(kubeadm安装方式)
/etc/kubernetes/manifests/kube-apiserver.yaml
apiVersion: v1
kind: Pod
metadata:
annotations:
kubeadm.kubernetes.io/kube-apiserver.advertise-address.endpoint: 10.150.90.242:6443
creationTimestamp: null
labels:
component: kube-apiserver
tier: control-plane
name: kube-apiserver
namespace: kube-system
spec:
containers:
- command:
- kube-apiserver
- --advertise-address=10.150.11.111
- --allow-privileged=true
- --authorization-mode=Node,RBAC
- --client-ca-file=/etc/kubernetes/pki/ca.crt
- --enable-admission-plugins=NodeRestriction
- --enable-bootstrap-token-auth=true
- --etcd-cafile=/etc/kubernetes/pki/etcd/ca.crt
- --etcd-certfile=/etc/kubernetes/pki/apiserver-etcd-client.crt
- --etcd-keyfile=/etc/kubernetes/pki/apiserver-etcd-client.key
- --etcd-servers=https://127.0.0.1:2379
- --kubelet-client-certificate=/etc/kubernetes/pki/apiserver-kubelet-client.crt
- --kubelet-client-key=/etc/kubernetes/pki/apiserver-kubelet-client.key
- --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname
- --proxy-client-cert-file=/etc/kubernetes/pki/front-proxy-client.crt
- --proxy-client-key-file=/etc/kubernetes/pki/front-proxy-client.key
- --requestheader-allowed-names=front-proxy-client
- --requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.crt
- --requestheader-extra-headers-prefix=X-Remote-Extra-
- --requestheader-group-headers=X-Remote-Group
- --requestheader-username-headers=X-Remote-User
- --secure-port=6443
- --service-account-key-file=/etc/kubernetes/pki/sa.pub
- --service-cluster-ip-range=10.96.0.0/12
- --tls-cert-file=/etc/kubernetes/pki/apiserver.crt
- --tls-private-key-file=/etc/kubernetes/pki/apiserver.key
- --audit-policy-file=/etc/kubernetes/audit-policy.yaml
- --audit-log-path=/var/log/kubernetes/audit/audit.log
image: registry.aliyuncs.com/google_containers/kube-apiserver:v1.20.0
imagePullPolicy: IfNotPresent
livenessProbe:
failureThreshold: 8
httpGet:
host: 10.150.90.242
path: /healthz
port: 6443
scheme: HTTPS
initialDelaySeconds: 15
timeoutSeconds: 15
name: kube-apiserver
resources:
requests:
cpu: 250m
volumeMounts:
- mountPath: /etc/ssl/certs
name: ca-certs
readOnly: true
- mountPath: /etc/pki
name: etc-pki
readOnly: true
- mountPath: /etc/kubernetes/pki
name: k8s-certs
readOnly: true
- mountPath: /etc/kubernetes/audit-policy.yaml
name: audit
readOnly: true
- mountPath: /var/log/kubernetes/audit/
name: audit-log
readOnly: false
hostNetwork: true
priorityClassName: system-cluster-critical
volumes:
- hostPath:
path: /etc/ssl/certs
type: DirectoryOrCreate
name: ca-certs
- hostPath:
path: /etc/pki
type: DirectoryOrCreate
name: etc-pki
- hostPath:
path: /etc/kubernetes/pki
type: DirectoryOrCreate
name: k8s-certs
- name: audit
hostPath:
path: /etc/kubernetes/audit-policy.yaml
type: File
- name: audit-log
hostPath:
path: /var/log/kubernetes/audit/
type: DirectoryOrCreate
status: {}
重启kube api-server 清单文件:(kubeadm安装方式)
systemctl restart kubelet
参考:
https://www.kubernetes.org.cn/1031.html
https://www.kubernetes.org.cn/1090.html
https://www.kubernetes.org.cn/1195.html
https://kubernetes.io/zh/docs/tasks/debug-application-cluster/audit/
https://kubernetes.io/zh/docs/reference/command-line-tools-reference/kube-apiserver/
https://kubernetes.io/zh/docs/reference/config-api/apiserver-audit.v1/#audit-k8s-io-v1-Event
https://www.kubernetes.org.cn/2611.html?spm=a2c6h.12873639.0.0.616c3b5fuDXKll
https://developer.aliyun.com/article/686982
最新文章
- Java字节数组转按radix进制输出
- C++设计模式-Mediator中介者模式
- Lucene.net 多条件查询搜索
- hdu 5327 Olympiad
- 如何用 redis 造一把分布式锁
- iOS开发——动画篇Swift篇&;动画效果的实现
- 广州麒麟网络工作室 qlgame eninge(anroid) opengles c++ matrix
- 分享一个 CSDN 自动评论角本
- 理解 Python 中的线程
- 不再害羞,过程比结果更重要;分享一套 CodeSmit 代码生成模板。
- mariadb cache
- iOS开发——判断邮箱格式
- Docker Registry
- sql集锦
- python之面试复习
- 【easy】88. Merge Sorted Array 合并两个有序数组
- 主席树入门——询问区间第k大pos2104,询问区间<;=k的元素个数hdu4417
- 使用curl自动签到百度贴吧
- Mac 终端添加代码到SVN
- Delphi2010如何获取程序内部自身版本号?
热门文章
- Python全栈学习笔记---正则表达式(二)
- sync.Once
- [菜鸡随笔-2]用mv代替rm安全地进行删除操作的小技巧
- TP3.2.x判断手机端访问,同一个域名在PC和手机端展示不同模板(半独立式网站)
- 前端之Vue day07 混入、插件、elementui、Router、Vuex
- 软件工程日报三——创建一个简单app和SDK的介绍
- spring mvc @Configuration addConverterFactory 无效问题
- JAVA学习笔记-08
- golang yaml配置
- 解决habbybase 操作hbase报错TTransportException(type=4,message=’TSocket read 0 bytes)