一、部署Kubelet

1.1 集群规划

主机名		 角色		IP
hdss7-21 kubelet 10.4.7.21
hdss7-22 kubelet 10.4.7.22

注意:部署以10.4.7.21为例,10.4.7.22节点类似

1.2 签发kubelet证书

证书签发需要在10.4.7.200上操作

[root@hdss7-200 ~]# cd /opt/certs/
注意:将所有可能的kubelet服务器的IP都加进去,后期如果需要再加入其他IP节点的话就需要重新签发此证书,有计划的将证书替换成最新的,最好避免后期加入新的节点。
[root@hdss7-200 certs]# vim kubelet-csr.json
{
"CN": "k8s-kubelet",
"hosts": [
"127.0.0.1",
"10.4.7.10",
"10.4.7.21",
"10.4.7.22",
"10.4.7.23",
"10.4.7.24",
"10.4.7.25",
"10.4.7.26",
"10.4.7.27",
"10.4.7.28"
],
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"ST": "beijing",
"L": "beijing",
"O": "od",
"OU": "ops"
}
]
}
生成证书
[root@hdss7-200 certs]# cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=server kubelet-csr.json | cfssl-json -bare kubelet
注意私钥文件的属性权限是600
certs]# ll kubelet*
-rw-r--r-- 1 root root 1115 6月 10 00:04 kubelet.csr
-rw-r--r-- 1 root root 452 6月 10 00:03 kubelet-csr.json
-rw------- 1 root root 1675 6月 10 00:04 kubelet-key.pem
-rw-r--r-- 1 root root 1468 6月 10 00:04 kubelet.pem
分发证书
[root@hdss7-200 certs]# scp kubelet.pem kubelet-key.pem hdss7-21:/opt/kubernetes/server/bin/certs/
[root@hdss7-200 certs]# scp kubelet.pem kubelet-key.pem hdss7-22:/opt/kubernetes/server/bin/certs/

1.3 创建kubelet的配置

在10.4.7.21,22服务器上操作

1.3.1 set-cluster:创建需要连接的集群信息,可以创建多个k8s信息(会将ca.pem证书编码后嵌入到/opt/kubernetes/conf/kubelet.kubeconfig配置文件中)

注意:10.4.7.10是apiserver的VIP,之前我们在10.4.7.11/21上部署的nginx就是代理10.4.7.21/22的apiserver集群,部署的keepalived的VIP就是10.4.7.10

[root@hdss7-21 ~]# cd /opt/kubernetes/
[root@hdss7-21 conf]# kubectl config set-cluster myk8s \
--certificate-authority=/opt/kubernetes/server/bin/certs/ca.pem \
--embed-certs=true \
--server=https://10.4.7.10:7443 \
--kubeconfig=/opt/kubernetes/conf/kubelet.kubeconfig
Cluster "myk8s" set.
[root@hdss7-21 conf]# ll /opt/kubernetes/conf/
总用量 8
-rw-r--r-- 1 root root 2223 6月 8 22:00 audit.yaml
-rw------- 1 root root 1986 6月 10 00:14 kubelet.kubeconfig

1.3.2 set-credentials:创建用户账号,即用户登录的客户端私有证书,可以创建多个证书(将client.pem证书和client-key.pem私钥编码后嵌入到kubelet.kubeconfig文件中)

[root@hdss7-21 conf]# kubectl config set-credentials k8s-node \
--client-certificate=/opt/kubernetes/server/bin/certs/client.pem \
--client-key=/opt/kubernetes/server/bin/certs/client-key.pem \
--embed-certs=true \
--kubeconfig=/opt/kubernetes/conf/kubelet.kubeconfig
User "k8s-node" set.

1.3.3 set-context:设置context,即确定账号和集群对应关系

[root@hdss7-21 conf]# kubectl config set-context myk8s-context \
--cluster=myk8s \
--user=k8s-node \
--kubeconfig=/opt/kubernetes/conf/kubelet.kubeconfig
Context "myk8s-context" created.

1.3.4 use-context:设置当前使用哪个context

[root@hdss7-21 conf]# kubectl config use-context myk8s-context --kubeconfig=/opt/kubernetes/conf/kubelet.kubeconfig
Switched to context "myk8s-context".
把此配置传送给10.4.7.22,那么在22上就不需要重复操作以上4个步骤了
[root@hdss7-21 conf]# scp /opt/kubernetes/conf/kubelet.kubeconfig hdss7-22:/opt/kubernetes/conf/

1.4 授权k8s-node用户

此步骤只需要在一台master节点上操作就行(10.4.7.21)

授权k8s-node用户绑定集群角色system:node,让k8s-node拥有具备运算节点的权限

[root@hdss7-21 conf]# vim /opt/kubernetes/conf/k8s-node.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: k8s-node
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: system:node
subjects:
- apiGroup: rbac.authorization.k8s.io
kind: User
name: k8s-node
[root@hdss7-21 conf]# kubectl create -f /opt/kubernetes/conf/k8s-node.yaml
clusterrolebinding.rbac.authorization.k8s.io/k8s-node created
创建资源(会存储到etcd中)
[root@hdss7-21 conf]# kubectl get clusterrolebinding k8s-node
NAME AGE
k8s-node 51s
注意:查看7443端口是否正常启动,非常重要,7443端口无法连接会导致node节点无法加入到master节点
~]# telnet 10.4.7.10 7443
Trying 10.4.7.10...
Connected to 10.4.7.10.
Escape character is '^]'.
^]
telnet> q

删除资源命令如下

[root@hdss7-21 conf]# kubectl delete -f k8s-node.yaml
clusterrolebinding.rbac.authorization.k8s.io "k8s-node" deleted

查看资源配置

[root@hdss7-21 conf]# kubectl get clusterrolebinding k8s-node -o yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
creationTimestamp: "2021-06-10T13:51:06Z"
name: k8s-node
resourceVersion: "12725"
selfLink: /apis/rbac.authorization.k8s.io/v1/clusterrolebindings/k8s-node
uid: e70f91af-c9f2-11eb-aaf3-000c29e396b1
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: system:node
subjects:
- apiGroup: rbac.authorization.k8s.io
kind: User
name: k8s-node

1.5 准备pause基础镜像

因为kubelet在启动容器时需要有一个基础镜像初始化网络空间等来帮助我们去启动容器,从而让我们能够启动pod;

将pause镜像放入到harbor私有仓库中,仅在10.4.7.200上操作,确保harbor和docker运行正常,提前检查

下载镜像
[root@hdss7-200 ~]# docker image pull kubernetes/pause
打标签
[root@hdss7-200 ~]# docker image tag kubernetes/pause:latest harbor.od.com/public/pause:latest
登录harbor
[root@hdss7-200 ~]# docker login -u admin harbor.od.com
推送pause镜像到harbor私有仓库
[root@hdss7-200 ~]# docker image push harbor.od.com/public/pause:latest

1.6 创建kubelet启动脚本

在node节点创建启动脚本,并启动kubelet,在10.4.7.21/22上操作,以21为例

22上修改--hostname-override项

[root@hdss7-21 ~]# vim /opt/kubernetes/server/bin/kubelet-startup.sh
#!/bin/sh WORK_DIR=$(dirname $(readlink -f $0))
[ $? -eq 0 ] && cd $WORK_DIR || exit /opt/kubernetes/server/bin/kubelet \
--anonymous-auth=false \
--cgroup-driver systemd \
--cluster-dns 192.168.0.2 \
--cluster-domain cluster.local \
--runtime-cgroups=/systemd/system.slice \
--kubelet-cgroups=/systemd/system.slice \
--fail-swap-on="false" \
--client-ca-file ./certs/ca.pem \
--tls-cert-file ./certs/kubelet.pem \
--tls-private-key-file ./certs/kubelet-key.pem \
--hostname-override hdss7-21.host.com \
--image-gc-high-threshold 20 \
--image-gc-low-threshold 10 \
--kubeconfig ../../conf/kubelet.kubeconfig \
--log-dir /data/logs/kubernetes/kube-kubelet \
--pod-infra-container-image harbor.od.com/public/pause:latest \
--root-dir /data/kubelet

1.7 添加权限,创建目录

[root@hdss7-21 ~]# chmod +x /opt/kubernetes/server/bin/kubelet-startup.sh
[root@hdss7-21 ~]# mkdir -p /data/logs/kubernetes/kube-kubelet /data/kubelet

1.8 配置supervisor配置

[root@hdss7-21 ~]# vim /etc/supervisord.d/kube-kubelet.ini
[program:kube-kubelet-7-21]
command=/opt/kubernetes/server/bin/kubelet-startup.sh
numprocs=1
directory=/opt/kubernetes/server/bin
autostart=true
autorestart=true
startsecs=30
startretries=3
exitcodes=0,2
stopsignal=QUIT
stopwaitsecs=10
user=root
redirect_stderr=true
stdout_logfile=/data/logs/kubernetes/kube-kubelet/kubelet.stdout.log
stdout_logfile_maxbytes=64MB
stdout_logfile_backups=5
stdout_capture_maxbytes=1MB
stdout_events_enabled=false

1.9 启动服务并检查

[root@hdss7-21 ~]# supervisorctl update
kube-kubelet-7-21: added process group
[root@hdss7-21 ~]# supervisorctl status
etcd-server-7-21 RUNNING pid 1172, uptime 1:06:46
kube-apiserver-7-21 RUNNING pid 1183, uptime 1:06:46
kube-controller-manager-7-21 RUNNING pid 1167, uptime 1:06:46
kube-kubelet-7-21 RUNNING pid 2280, uptime 0:01:44
kube-scheduler-7-21 RUNNING pid 1169, uptime 1:06:46
[root@hdss7-22 ~]# tail -100f /data/logs/kubernetes/kube-kubelet/kubelet.stdout.log
I0713 21:44:08.453953 2265 kubelet_node_status.go:75] Successfully registered node hdss7-21.host.com
I0713 21:44:08.509328 2265 cpu_manager.go:155] [cpumanager] starting with none policy
I0713 21:44:08.509382 2265 cpu_manager.go:156] [cpumanager] reconciling every 10s
I0713 21:44:08.509441 2265 policy_none.go:42] [cpumanager] none policy: Start
W0713 21:44:08.644794 2265 manager.go:540] Failed to retrieve checkpoint for "kubelet_internal_checkpoint": checkpoint is not found
I0713 21:44:08.878478 2265 reconciler.go:154] Reconciler: start to sync state
出现如上表示正常启动
查看node节点是否加入到集群中
[root@hdss7-21 ~]# kubectl get node
NAME STATUS ROLES AGE VERSION
hdss7-21.host.com Ready <none> 13s v1.14.10
hdss7-22.host.com NotReady <none> 0s v1.14.10
别急,需要加载一会
[root@hdss7-21 ~]# kubectl get node
NAME STATUS ROLES AGE VERSION
hdss7-21.host.com Ready <none> 55m v1.14.10
hdss7-22.host.com Ready <none> 54m v1.14.10

1.10 修改节点角色

[root@hdss7-21 ~]# kubectl label node hdss7-21.host.com node-role.kubernetes.io/node=
node/hdss7-21.host.com labeled
[root@hdss7-21 ~]# kubectl label node hdss7-21.host.com node-role.kubernetes.io/master=
node/hdss7-21.host.com labeled
[root@hdss7-21 ~]# kubectl label node hdss7-22.host.com node-role.kubernetes.io/master=
node/hdss7-22.host.com labeled
[root@hdss7-21 ~]# kubectl label node hdss7-22.host.com node-role.kubernetes.io/node=
node/hdss7-22.host.com labeled
[root@hdss7-21 ~]# kubectl get node
NAME STATUS ROLES AGE VERSION
hdss7-21.host.com Ready master,node 57m v1.14.10
hdss7-22.host.com Ready master,node 57m v1.14.10

1.11 安装部署其他节点

在10.4.7.22上同样操作

1.12 报错排查

在10.4.7.21(10.4.7.10)master节点上查看node,发现无任何资源可访问,如下

[root@hdss7-21 ~]#  kubectl get node
No resources found.

查看kubectl日志

[root@hdss7-21 ~]# tail -100f /data/logs/kubernetes/kube-kubelet/kubelet.stdout.log

第一种如下:

failed to ensure node lease exists connect: no route to host
原因:这是根本就没有10.4.7.10这个ip,或者无法连接到此ip,ping一下,检查是否可以连接,添加此虚IP重新,重新执行本章1.2步骤即可

第二种如下:

E0611 20:41:55.908234    1414 kubelet.go:2246] node "hdss7-22.host.com" not found
E0611 20:41:56.008667 1414 kubelet.go:2246] node "hdss7-22.host.com" not found
这个报错可以忽略

第三种如下:

E0611 20:41:55.838167    1414 reflector.go:126] k8s.io/client-go/informers/factory.go:133: Failed to list *v1beta1.RuntimeClass: Get https://10.4.7.10:7443/apis/node.k8s.io/v1beta1/runtimeclasses?limit=500&resourceVersion=0: dial tcp 10.4.7.10:7443: connect: connection refused
这就说明虽然找到了这个服务器,但是拒绝连接,测试一下端口的连通性,发现被拒绝,无此端口
[root@hdss7-21 ~]# telnet 10.4.7.10 7443
Trying 10.4.7.10...
telnet: connect to address 10.4.7.10: Connection refused
检查虚IP是否正确配置,检查1.2步骤是否正确执行,在本地telnet一下,是否正确启动7443端口,正常启动后重启一下kube-kubelet-7-21,kube-kubelet-7-22服务即可
正确的日志如下:
I0611 21:06:18.917499 9153 kubelet_node_status.go:72] Attempting to register node hdss7-22.host.com
I0611 21:06:18.947122 9153 kubelet_node_status.go:75] Successfully registered node hdss7-22.host.com
I0611 21:06:18.989477 9153 kubelet.go:1825] skipping pod synchronization - container runtime status check may not have completed yet.
I0611 21:06:19.015529 9153 cpu_manager.go:155] [cpumanager] starting with none policy
I0611 21:06:19.015565 9153 cpu_manager.go:156] [cpumanager] reconciling every 10s

二、部署kube-proxy

Kube-proxy实际上是维护了pod网络、节点(node)网络与cluster(service)网络三者之间的关系

2.1 集群规划

注意:部署以10.4.7.21为例,22节点部署类似

主机名			角色			IP
hdss7-21 kube-proxy 10.4.7.21
hdss7-22 kube-proxy 10.4.7.22

2.2 签发kube-proxy证书

在10.4.7.200证书签发服务器上操作

创建签发证书的请求文件
[root@hdss7-200 ~]# cd /opt/certs/
[root@hdss7-200 certs]# vim kube-proxy-csr.json
{
"CN": "system:kube-proxy",
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"ST": "beijing",
"L": "beijing",
"O": "od",
"OU": "ops"
}
]
}
[root@hdss7-200 certs]# cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=client kube-proxy-csr.json |cfssl-json -bare kube-proxy-client
certs]# ll kube-proxy*
-rw-r--r-- 1 root root 1005 6月 12 20:49 kube-proxy-client.csr
-rw------- 1 root root 1679 6月 12 20:49 kube-proxy-client-key.pem
-rw-r--r-- 1 root root 1375 6月 12 20:49 kube-proxy-client.pem
-rw-r--r-- 1 root root 267 6月 12 20:49 kube-proxy-csr.json
分发证书
[root@hdss7-200 certs]# scp kube-proxy-client-key.pem kube-proxy-client.pem hdss7-21:/opt/kubernetes/server/bin/certs/
[root@hdss7-200 certs]# scp kube-proxy-client-key.pem kube-proxy-client.pem hdss7-22:/opt/kubernetes/server/bin/certs/

2.3 创建kube-proxy配置

在所有node节点部署,涉及服务器10.4.7.21,22,步骤与创建kubelet相同

[root@hdss7-21 ~]# kubectl config set-cluster myk8s \
--certificate-authority=/opt/kubernetes/server/bin/certs/ca.pem \
--embed-certs=true \
--server=https://10.4.7.10:7443 \
--kubeconfig=/opt/kubernetes/conf/kube-proxy.kubeconfig
Cluster "myk8s" set.
[root@hdss7-21 ~]# kubectl config set-credentials kube-proxy \
--client-certificate=/opt/kubernetes/server/bin/certs/kube-proxy-client.pem \
--client-key=/opt/kubernetes/server/bin/certs/kube-proxy-client-key.pem \
--embed-certs=true \
--kubeconfig=/opt/kubernetes/conf/kube-proxy.kubeconfig
User "kube-proxy" set.
[root@hdss7-21 ~]# kubectl config set-context myk8s-context \
--cluster=myk8s \
--user=kube-proxy \
--kubeconfig=/opt/kubernetes/conf/kube-proxy.kubeconfig
Context "myk8s-context" created.
[root@hdss7-21 ~]# kubectl config use-context myk8s-context --kubeconfig=/opt/kubernetes/conf/kube-proxy.kubeconfig
Switched to context "myk8s-context".
传送配置,22上就不需要操作了
[root@hdss7-21 ~]# scp /opt/kubernetes/conf/kube-proxy.kubeconfig hdss7-22:/opt/kubernetes/conf/

2.4 加载IPvs模块

kube-proxy共有三种流量调度模式,分别是userspace,iptables和ipvs,目前ipvs是最匹配的

在21和22上操作

查看现有的ipvs模块
[root@hdss7-21 ~]# lsmod | grep ip_vs
加载ipvs模块
[root@hdss7-21 ~]# for i in $(ls /usr/lib/modules/$(uname -r)/kernel/net/netfilter/ipvs|grep -o "^[^.]*");do echo $i; /sbin/modinfo -F filename $i >/dev/null 2>&1 && /sbin/modprobe $i;done
ip_vs_dh
ip_vs_ftp
ip_vs
ip_vs_lblc
ip_vs_lblcr
ip_vs_lc
ip_vs_nq
ip_vs_pe_sip
ip_vs_rr
ip_vs_sed
ip_vs_sh
ip_vs_wlc
ip_vs_wrr
查看
[root@hdss7-21 ~]# lsmod | grep ip_vs
ip_vs_wrr 12697 0
ip_vs_wlc 12519 0
ip_vs_sh 12688 0
ip_vs_sed 12519 0
ip_vs_rr 12600 0
ip_vs_pe_sip 12740 0
nf_conntrack_sip 33780 1 ip_vs_pe_sip
ip_vs_nq 12516 0
ip_vs_lc 12516 0
ip_vs_lblcr 12922 0
ip_vs_lblc 12819 0
ip_vs_ftp 13079 0
ip_vs_dh 12688 0
nf_nat 26583 3 ip_vs_ftp,nf_nat_ipv4,nf_nat_masquerade_ipv4
ip_vs 145497 24 ip_vs_dh,ip_vs_lc,ip_vs_nq,ip_vs_rr,ip_vs_sh,ip_vs_ftp,ip_vs_sed,ip_vs_wlc,ip_vs_wrr,ip_vs_pe_sip,ip_vs_lblcr,ip_vs_lblc
nf_conntrack 139224 8 ip_vs,nf_nat,nf_nat_ipv4,xt_conntrack,nf_nat_masquerade_ipv4,nf_conntrack_netlink,nf_conntrack_sip,nf_conntrack_ipv4
libcrc32c 12644 3 ip_vs,nf_nat,nf_conntrack

2.5 创建kube-proxy启动脚本

此操作需要在21和22上进行操作

--hostname-override需要修改为本机的主机名

[root@hdss7-21 ~]# vim /opt/kubernetes/server/bin/kube-proxy-startup.sh
#!/bin/sh WORK_DIR=$(dirname $(readlink -f $0))
[ $? -eq 0 ] && cd $WORK_DIR || exit /opt/kubernetes/server/bin/kube-proxy \
--cluster-cidr 172.7.0.0/16 \
--hostname-override hdss7-21.host.com \
--proxy-mode=ipvs \
--ipvs-scheduler=nq \
--kubeconfig ../../conf/kube-proxy.kubeconfig

2.6 设置权限,创建目录

[root@hdss7-21 ~]# chmod +x /opt/kubernetes/server/bin/kube-proxy-startup.sh
[root@hdss7-21 ~]# mkdir -p /data/logs/kubernetes/kube-proxy

2.7 创建supervisor配置

[root@hdss7-21 ~]# vim /etc/supervisord.d/kube-proxy.ini
[program:kube-proxy-7-21]
command=/opt/kubernetes/server/bin/kube-proxy-startup.sh
numprocs=1
directory=/opt/kubernetes/server/bin
autostart=true
autorestart=true
startsecs=30
startretries=3
exitcodes=0,2
stopsignal=QUIT
stopwaitsecs=10
user=root
redirect_stderr=true
stdout_logfile=/data/logs/kubernetes/kube-proxy/proxy.stdout.log
stdout_logfile_maxbytes=64MB
stdout_logfile_backups=5
stdout_capture_maxbytes=1MB
stdout_events_enabled=false

2.8 启动服务并检查

[root@hdss7-21 ~]# supervisorctl update
kube-proxy-7-21: added process group
查看代理状态
[root@hdss7-21 ~]# supervisorctl status
etcd-server-7-21 RUNNING pid 1319, uptime 0:48:02
kube-apiserver-7-21 RUNNING pid 1328, uptime 0:48:02
kube-controller-manager-7-21 RUNNING pid 1305, uptime 0:48:02
kube-kubelet-7-21 RUNNING pid 1308, uptime 0:48:02
kube-proxy-7-21 RUNNING pid 11663, uptime 0:02:10
kube-scheduler-7-21 RUNNING pid 1316, uptime 0:48:02
[root@hdss7-21 ~]# yum -y install ipvsadm
查看ipvs代理信息,可以看到192.168.0.1:443端口代理了10.4.7.21/22:6443端口
[root@hdss7-21 ~]# ipvsadm -Ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP 192.168.0.1:443 nq
-> 10.4.7.21:6443 Masq 1 0 0
-> 10.4.7.22:6443 Masq 1 0 0
查看service的信息
[root@hdss7-21 ~]# kubectl get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes ClusterIP 192.168.0.1 <none> 443/TCP 3d23h

2.9 安装部署集群其他节点

部署集群其他节点

最新文章

  1. Android targetSdkVersion 对生命周期的影响
  2. 【WEB前端】CSS常用选择器
  3. KVM虚拟机网络基础及优化说明
  4. dedecms头部标签(标题,关键词,描述标签)(借用)
  5. webapp在Android中点击链接的时候会有淡蓝色的遮罩层
  6. 使用 HTML5 input 类型提升移动端输入体验(键盘)
  7. struts2.0 s标签_小小鸟_百度空间
  8. 《javascript语言精粹》——第4章函数
  9. 关于&quot;模块计算机类型与目标计算机类型冲突&quot;的解决
  10. 201521123044 《Java程序设计》第01周学习总结
  11. 《共享库PATH与ld.so.conf简析》
  12. flask sqlchemy 多对多的自引用关系定义
  13. python学习日记(2/3区别,环境,变量,数据类型以及简单习题)
  14. [转] React 是什么
  15. redis的高级事务CAS(乐观锁)
  16. LeetCode--204--计数质数
  17. springboot +element-axios跨域请求
  18. clone一行div tr 每次增量赋值
  19. JS-DOM-随时更新
  20. MySQL-安全对调两个表名

热门文章

  1. 训练一个图像分类器demo in PyTorch【学习笔记】
  2. 如何搭建android源代码repo仓库
  3. Spring框架系列(14) - SpringMVC实现原理之DispatcherServlet处理请求的过程
  4. 最优化:凸集、凸函数、KKT条件极其解释
  5. elementplus轮播图初始空白
  6. 迭代器和增强for循环
  7. Java开发学习(十八)----AOP通知获取数据(参数、返回值、异常)
  8. kubernetes之DaemonSet以及滚动更新
  9. Vue路由器的hash和history两种工作模式 &amp;&amp; Vue项目编译部署
  10. LuoguP3690 【模板】Link Cut Tree (LCT)