centos7之使用最新版的kubeadm体验k8s1.12.0

1、环境准备

centos7 、docker-ce18.06.1-ce、kubeadm、kubelet、kubectl

2、安装

yum安装，准备repo文件

docker：

[docker-ce-stable]

name=Docker CE Stable - $basearch

baseurl=https://download.docker.com/linux/centos/7/$basearch/stable

enabled=

gpgcheck=

gpgkey=https://download.docker.com/linux/centos/gpg

[docker-ce-stable-debuginfo]

name=Docker CE Stable - Debuginfo $basearch

baseurl=https://download.docker.com/linux/centos/7/debug-$basearch/stable

enabled=

gpgcheck=

gpgkey=https://download.docker.com/linux/centos/gpg

[docker-ce-stable-source]

name=Docker CE Stable - Sources

baseurl=https://download.docker.com/linux/centos/7/source/stable

enabled=

gpgcheck=

gpgkey=https://download.docker.com/linux/centos/gpg

kubeadm、kubelet、kubctl

[kubernetes]

name=Kubernetes

baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64

enabled=

gpgcheck=

repo_gpgcheck=

gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg

yum -y install docker-ce-18.06..ce-.el7.x86_64

yum -y install  kubelet-1.12.2-0.x86_64  kubectl-1.12.2-0.x86_64 kubeadm-1.12.2-0.x86_64

3、配置docker

#vim /usr/lib/systemd/system/docker.service

 ExecStart=/usr/bin/dockerd --graph=/data/docker --storage-driver=overlay2

# mkdir -p /etc/systemd/system/docker.service.d

#vim http-proxy.conf

[Service]

Environment="HTTP_PROXY=http://10.10.23.74:8118" "NO_PROXY=localhost,127.0.0.1,0.0.0.0,10.10.29.43,10.10.25.49,172.11.0.0,172.10.0.0,172.11.0.0/16,172.10.0.0/16,10.,172.,.evo.get.com,.kube.hpp.com,charts.gitlab.io,.mirror.ucloud.cn"

#cat https-proxy.conf
[Service]

Environment="HTTPS_PROXY=http://10.10.23.74:8118" "NO_PROXY=localhost,127.0.0.1,0.0.0.0,10.10.29.43,10.10.25.49,172.11.0.0,172.10.0.0,172.11.0.0/16,172.10.0.0/16,10.,172.,.evo.get.com,.kube.hpp.com,charts.gitlab.io,.mirror.ucloud.cn"

shadowsocks的安装参考我的另外的一篇博客：https://www.cnblogs.com/cuishuai/p/8463458.html

4、初始化

创建/etc/sysctl.d/k8s.conf文件

cat /etc/sysctl.d/k8s.conf

net.bridge.bridge-nf-call-ip6tables =

net.bridge.bridge-nf-call-iptables =

net.ipv4.ip_forward =

vm.swappiness=0

sysctl -p  /etc/sysctl.d/k8s.conf

swapoff -a 
systemctl enable kubelet
systemctl start kubelet
kubeadm init  --kubernetes-version=v1.12.0 --pod-network-cidr=10.244.0.0/  --apiserver-advertise-address=192.18.1.12

--apiserver-advertise-address

是master的apiserver的监听地址，默认是本机ip。

 mkdir -p $HOME/.kube

 sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config

 sudo chown $(id -u):$(id -g) $HOME/.kube/config

默认master节点是node-role.kubernetes.io/master:NoSchedule，需要做一个修改临时的，为了测试和后面部署flannel。

kubectl taint nodes ku node-role.kubernetes.io/master-

ku是我的master的节点名称。当然可以制定为--all。

安装flannel

wget https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml

vim kube-flannel.yml

   args:

        - --ip-masq

        - --kube-subnet-mgr

        - --iface=eth0

kubectl apply -f kube-flannel.yml

由于前面已经修改了master的taint，可以直接部署，如果没有修改的话，可以修改flannel的压马路文件来实现部署，否则会部署失败。

修改

spec:
hostNetwork: true
nodeSelector:
beta.kubernetes.io/arch: amd64
tolerations:
- operator: Exists
effect: NoSchedule

tolerations:

      - key: node-role.kubernetes.io/master

        operator: Exists

        effect: NoSchedule

      - key: node.kubernetes.io/not-ready

        operator: Exists

        effect: NoSchedule

测试DNS

kubectl run curl --image=radial/busyboxplus:curl -it

[ root@curl-5cc7b478b6-6cfqr:/ ]$ nslookup kubernetes.default

Server:    10.96.0.10

Address : 10.96.0.10 kube-dns.kube-system.svc.cluster.local

Name:      kubernetes.default

Address : 10.96.0.1 kubernetes.default.svc.cluster.local

进入pod

kubectl exec -it curl-5cc7b478b6-6cfqr -n default -- /bin/sh

！添加node节点到集群：

使用初始化得到的命令直接加入即可，node节点需要安装kubelet、kubeadm

kubeadm默认创建的token24小时失效，再次添加节点的时候，就会报错unauthorized。这里有两种方法：

1、直接生成一个永不过期的token（不推荐这种方式，线上集群的token还是要定时更换）

kubeadm  token create  --ttl

2、定时更换，设置一个合理的ttl，定时更换集群的token

# cat uptk.sh

#!/bin/bash

token=` kubeadm token list| awk '{print $1}' | grep -v TOKEN`

 for i in $token

 do

  kubeadm token delete $i

 done

kubeadm token create --ttl 72h

设置crontab，每两天更新一次，这个可以根据自己的需求做，因为我生成的token是72h，即3天。

crontab -e

* * */ * * /data/scripts/uptk.sh >& &

获取kubeadm join需要的信息：

1）token

kubeadm token list | grep authentication,signing | awk '{print $1}'

2）discovery-token-ca-cert-hash

openssl x509 -pubkey -in /etc/kubernetes/pki/ca.crt | openssl rsa -pubin -outform der >/dev/null | openssl dgst -sha256 -hex | sed 's/^.* //'

3）加入节点

kubeadm join --token c04f89.b781cdb55d83c1ef 10.10.3.4: --discovery-token-ca-cert-hash sha256:986e83a9cb948368ad0552b95232e31d3b76e2476b595bd1d905d5242ace29af

！集群移除节点

在master节点：

kubectl drain node2 --delete-local-data --force --ignore-daemonsets

kubectl delete node node2

node2是要删除的节点的名称。

在node2节点上执行：

kubeadm reset

ifconfig cni0 down

ip link delete cni0

ifconfig flannel. down

ip link delete flannel.

rm -rf /var/lib/cni/

安装helm

wget https://storage.googleapis.com/kubernetes-helm/helm-v2.11.0-linux-amd64.tar.gz

tar xf helm-v2.11.0-linux-amd64.tar.gz

cp  helm tiller  /usr/local/bin

创建tiller需要的用户，这里为了可以使用helm部署到所有的namespace里面，赋予clusterole权限，创建rbac-tiller.yaml

apiVersion: v1

kind: ServiceAccount

metadata:

  name: tiller

  namespace: kube-system

---

apiVersion: rbac.authorization.k8s.io/v1beta1

kind: ClusterRoleBinding

metadata:

  name: tiller

roleRef:

  apiGroup: rbac.authorization.k8s.io

  kind: ClusterRole

  name: cluster-admin

subjects:

  - kind: ServiceAccount

    name: tiller

    namespace: kube-system

kubectl  apply  -f  rbac-tiller.yaml

创建tiller部署在特定的namespace，并且helm部署的程序也在这个namespace里面：

参考：https://whmzsu.github.io/helm-doc-zh-cn/quickstart/rbac-zh_cn.html

创建namespace：

kubectl create namespace tiller-world

apiVersion: v1

kind: ServiceAccount

metadata:

  name: tiller

  namespace: tiller-world

---

apiVersion: rbac.authorization.k8s.io/v1beta1

kind: Role

metadata:

  name: tiller-manager

  namespace: tiller-world

rules:

- apiGroups: ["","extensions","apps"]

  resources: ["*"]

  verbs: ["*"]

---

apiVersion: rbac.authorization.k8s.io/v1beta1

kind: RoleBinding

metadata:

  name: tiller-binding

  namespace: tiller-world

subjects:

- kind: ServiceAccount

  name: tiller

  namespace: tiller-world

roleRef:

  kind: Role

  name: tiller-manager

  apiGroup: rbac.authorization.k8s.io

初始化：

helm init --service-account tiller  --upgrade

默认安装到kube-system空间下，可以自己指定namespace和image等。

如果将tiller安装到tiller-world下面则需要执行如下命令：

helm init --service-account tiller --tiller-namespace tiller-world  --upgrade

镜像：

# kubernetes

k8s.gcr.io/kube-apiserver:v1.12.0

k8s.gcr.io/kube-controller-manager:v1.12.0

k8s.gcr.io/kube-scheduler:v1.12.0

k8s.gcr.io/kube-proxy:v1.12.0

k8s.gcr.io/etcd:3.2.

k8s.gcr.io/pause:3.1

# network and dns

quay.io/coreos/flannel:v0.10.0-amd64

k8s.gcr.io/coredns:1.2.

# helm and tiller

gcr.io/kubernetes-helm/tiller:v2.11.0

# nginx ingress

quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.19.

k8s.gcr.io/defaultbackend:1.4

# dashboard and metric-sever

k8s.gcr.io/kubernetes-dashboard-amd64:v1.10.0

gcr.io/google_containers/metrics-server-amd64:v0.3.0

1.12.1版本也测试过了，但是启动coredns的时候总是报错，起不来。

// :: [INFO] plugin/reload: Running configuration MD5 = f65c4821c8a9b7b5eb30fa4fbc167769

// :: [FATAL] plugin/loop: Seen "HINFO IN 3256902131464476443.1309143030470211725." more than twice, loop detected

按照google上找到的方法在/etc/systemd/system/kubelet.service.d/10-kubeadm.conf添加如下内容：

Environment="KUBELET_KUBECONFIG_ARGS=--bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf --resolv-conf=/etc/resolv.conf"

并没有解决，看github上面的issue也是没有解决，有的是停掉systemd-resolved。但是是ubuntu上的。我是的是centos，感觉也是不好，于是回退到1.12.0没有出现那个问题。

巴特西

centos7之使用最新版的kubeadm体验k8s1.12.0

最新文章

热门文章