[Kubernetes](Node)Drain and Delete for node change

업데이트:

Purpose

서버의 물리적인 자원교체, 버전 업그레이드를 위해 클러스터에서 노드를 제외시키기 위함.


1. Node drain & delete

## drain
root@AJTV005 [~]kubectl drain ajtv009 --ignore-daemonsets
node/ajtv009 already cordoned
WARNING: ignoring DaemonSet-managed Pods: kube-system/kube-proxy-9bl98, kube-system/weave-net-bfs2f
evicting pod default/deploytest-79bdb557f6-fpl8c
pod/deploytest-79bdb557f6-fpl8c evicted
node/ajtv009 evicted

root@AJTV005 [~]kubectl get nodes -o wide
NAME      STATUS                     ROLES    AGE   VERSION   INTERNAL-IP    EXTERNAL-IP   OS-IMAGE                KERNEL-VERSION               CONTAINER-RUNTIME
ajtv005   Ready                      master   22h   v1.19.4   10.50.107.21   <none>        CentOS Linux 7 (Core)   3.10.0-1160.6.1.el7.x86_64   docker://19.3.14
ajtv006   Ready                      master   22h   v1.19.4   10.50.107.22   <none>        CentOS Linux 7 (Core)   3.10.0-1160.6.1.el7.x86_64   docker://19.3.14
ajtv007   Ready                      <none>   22h   v1.19.4   10.50.107.24   <none>        CentOS Linux 7 (Core)   3.10.0-1160.6.1.el7.x86_64   docker://19.3.14
ajtv008   Ready                      <none>   22h   v1.19.4   10.50.107.25   <none>        CentOS Linux 7 (Core)   3.10.0-1160.6.1.el7.x86_64   docker://19.3.14
ajtv009   Ready,SchedulingDisabled   master   22h   v1.19.4   10.50.107.26   <none>        CentOS Linux 7 (Core)   3.10.0-1160.6.1.el7.x86_64   docker://19.3.14

## delete node
root@AJTV005 [~]kubectl delete node ajtv009
node "ajtv009" deleted

root@AJTV005 [~]kubectl get node -o wide
NAME      STATUS   ROLES    AGE   VERSION   INTERNAL-IP    EXTERNAL-IP   OS-IMAGE                KERNEL-VERSION               CONTAINER-RUNTIME
ajtv005   Ready    master   22h   v1.19.4   10.50.107.21   <none>        CentOS Linux 7 (Core)   3.10.0-1160.6.1.el7.x86_64   docker://19.3.14
ajtv006   Ready    master   22h   v1.19.4   10.50.107.22   <none>        CentOS Linux 7 (Core)   3.10.0-1160.6.1.el7.x86_64   docker://19.3.14
ajtv007   Ready    <none>   22h   v1.19.4   10.50.107.24   <none>        CentOS Linux 7 (Core)   3.10.0-1160.6.1.el7.x86_64   docker://19.3.14
ajtv008   Ready    <none>   22h   v1.19.4   10.50.107.25   <none>        CentOS Linux 7 (Core)   3.10.0-1160.6.1.el7.x86_64   docker://19.3.14

2. kubeadm reset (삭제한 노드에서)

kubeadm reset

rm -rf /etc/cni/net.d
rm -rf $HOME/.kube/config

3. Get token, certs and hash key

3.1 Create token

root@AJTV005 [~]kubeadm token list
TOKEN                     TTL         EXPIRES                     USAGES                   DESCRIPTION                                                EXTRA GROUPS
v4f4is.ss7k5e1t27kgc46u   1h          2020-12-08T17:17:40+09:00   authentication,signing   The default bootstrap token generated by 'kubeadm init'.   system:bootstrappers:kubeadm:default-node-token
root@AJTV005 [~]kubeadm token delete v4f4is.ss7k5e1t27kgc46u
bootstrap token "v4f4is" deleted

root@AJTV005 [~]kubeadm token create
W1208 15:46:28.364740   28648 configset.go:348] WARNING: kubeadm cannot validate component configs for API groups [kubelet.config.k8s.io kubeproxy.config.k8s.io]
uljmut.h6sy5ibklt0d9vuh
root@AJTV005 [~]kubeadm token list
TOKEN                     TTL         EXPIRES                     USAGES                   DESCRIPTION                                                EXTRA GROUPS
uljmut.h6sy5ibklt0d9vuh   23h         2020-12-09T15:46:28+09:00   authentication,signing   <none>                                                     system:bootstrappers:kubeadm:default-node-token

token TTL 24h

3.2 Get The CA key hash

root@AJTV005 [~/scripts]openssl x509 -pubkey -in /etc/kubernetes/pki/ca.crt | openssl rsa -pubin -outform der 2>/dev/null | openssl dgst -sha256 -hex | sed 's/^.* //'
0f63b526211dacb46d50157ce99b53d6bfdc7246551e39baa7b47e396f97542a

3.3 인증서 생성 & 업로드

3.3.1 인증서 인증서 생성 & 업로드

## 인증서 확인
kubeadm alpha certs check-expiration

## 인증서 생성
root@AJTV005 [~]kubeadm alpha certs certificate-key
1fa779fa84a83eb6cc7f48e817928eff5690a06b3d4cc11682480e364b913091

## 인증서 업로드 
kubeadm init phase upload-certs --upload-certs --certificate-key=1fa779fa84a83eb6cc7f48e817928eff5690a06b3d4cc11682480e364b913091

TTL 2H

3.3.2 임의의값으로 인증서 생성 & 업로드

root@AJTV005 [~/scripts]kubeadm init phase upload-certs --upload-certs
W1209 00:57:53.829050    8487 configset.go:348] WARNING: kubeadm cannot validate component configs for API groups [kubelet.config.k8s.io kubeproxy.config.k8s.io]
[upload-certs] Storing the certificates in Secret "kubeadm-certs" in the "kube-system" Namespace
[upload-certs] Using certificate key:
12d076d4733416c12c491ac54f929d43b4b0f721b044d0202b291d20f1656b96

4. Join with token

For Master

kubeadm join 10.50.107.23:8443 --token uljmut.h6sy5ibklt0d9vuh --discovery-token-ca-cert-hash sha256:0f63b526211dacb46d50157ce99b53d6bfdc7246551e39baa7b47e396f97542a     --control-plane --certificate-key 1fa779fa84a83eb6cc7f48e817928eff5690a06b3d4cc11682480e364b913091 --v=5

kubeadm join 10.50.107.23:8443 --token hbqmn6.4bu4lp8ik046qy78 \
--discovery-token-ca-cert-hash sha256:0f63b526211dacb46d50157ce99b53d6bfdc7246551e39baa7b47e396f97542a \
--control-plane \
--certificate-key 07a03068518c444d582123cb0fe38ae217cabbca521d8590667e8ab64322a8a9

For Node

kubeadm join 10.50.107.23:8443 --token uljmut.h6sy5ibklt0d9vuh     --discovery-token-ca-cert-hash sha256:0f63b526211dacb46d50157ce99b53d6bfdc7246551e39baa7b47e396f97542a

Troubleshooting

Remove master node from a HA Cluster and also from cluster

아래 작업 없이는 HA 구성된 Master Node 교체가 안됨 ( Worker Node는 상관 없음 )

Get member list

## ETCDCTL_AP3 etcdctl member list 명령어로도 획득 가능

root@AJTV005 [/]kubectl exec etcd-ajtv005 -n kube-system -- etcdctl --cacert /etc/kubernetes/pki/etcd/ca.crt --cert /etc/kubernetes/pki/etcd/peer.crt --key /etc/kubernetes/pki/etcd/peer.key member list
38e227bede457131, started, ajtv009, https://10.50.107.26:2380, https://10.50.107.26:2379, false
5f05adedb10bbff4, started, ajtv006, https://10.50.107.22:2380, https://10.50.107.22:2379, false
a8e5615362288545, started, ajtv005, https://10.50.107.21:2380, https://10.50.107.21:2379, false

Remove member

root@AJTV005 [/]kubectl exec etcd-ajtv005 -n kube-system -- etcdctl --cacert /etc/kubernetes/pki/etcd/ca.crt --cert /etc/kubernetes/pki/etcd/peer.crt --key /etc/kubernetes/pki/etcd/peer.key member remove 38e227bede457131
Member 38e227bede457131 removed from cluster ee7be35e4ed61075

댓글남기기