虚拟机部署k8s集群指南

记录一下用VMware虚拟机搭建k8s集群的过程，参考了云原生Java架构师的第一课K8s+Docker+KubeSphere+DevOps和VMware虚拟机部署k8s集群。

安装虚拟机

虚拟机安装CentOS7操作系统步骤可参考文章VMware虚拟机部署k8s集群，重点在于网络配置。

注意事项：

1.虚拟机内存至少要设置成4GB，否则可能不够用

2.网络类型选择 使用网络地址转换(NAT)(E) 选项

虚拟机网络配置

k8s master节点IP：172.31.0.3
k8s worker1节点IP：172.31.0.4
k8s worker2节点IP：172.31.0.5
子网掩码：255.255.0.0
网关：172.31.0.2
DNS：114.114.114.114

• 重要步骤1：VMware设置虚拟机网络

  点击主页–>编辑–>虚拟网络编辑器，按照下方图片设置：

• 重要步骤2：centos7虚拟机设置网络

  安装centos7系统时，点击网络和主机名–>配置，按照下方图片设置：

虚拟机安装Docker

参考文章Docker安装指南

部署k8s集群

kubeadm部署集群

安装kubeadm

**注意事项(摘自k8s官方文档)**：

1. 一台兼容的 Linux 主机。Kubernetes 项目为基于 Debian 和 Red Hat 的 Linux 发行版以及一些不提供包管理器的发行版提供通用的指令

2. 每台机器 2 GB 或更多的 RAM （如果少于这个数字将会影响你应用的运行内存）

3. 2 CPU 核或更多

4. 集群中的所有机器的网络彼此均能相互连接(公网和内网都可以)

5. 节点之中不可以有重复的主机名、MAC 地址或 product_uuid。请参见这里了解更多详细信息。

6. 开启机器上的某些端口。请参见这里 了解更多详细信息。

7. 禁用交换分区。为了保证 kubelet 正常工作，你 必须 禁用交换分区。

基础环境准备

# 将 SELinux 设置为 permissive 模式（相当于将其禁用）
sudo setenforce 0 # 临时性
sudo sed -i 's/^SELINUX=enforcing$/SELINUX=permissive/' /etc/selinux/config # 永久性

# 关闭swap
swapoff -a # 临时性
sed -ri 's/.*swap.*/#&/' /etc/fstab # 永久性

# 关闭防火墙
systemctl stop firewalld # 临时性
systemctl disable firewalld # 永久性

#允许 iptables 检查桥接流量
cat <<EOF | sudo tee /etc/modules-load.d/k8s.conf
br_netfilter
EOF

cat <<EOF | sudo tee /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
EOF

# 让以上配置生效
sudo sysctl --system

安装kubelet、kubeadm、kubectl

# 配置k8s下载的地址
cat <<EOF | sudo tee /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=http://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64
enabled=1
gpgcheck=0
repo_gpgcheck=0
gpgkey=http://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg
   http://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
exclude=kubelet kubeadm kubectl
EOF

# 安装3大件(最好不要安装1.24.x版本的kubenetes，因为Docker Shim在该版本之后就被移除了)
sudo yum install -y kubelet-1.22.11 kubeadm-1.22.11 kubectl-1.22.11 --disableexcludes=kubernetes

# 启动kubelet
sudo systemctl enable --now kubelet

构建集群

下载镜像

# 查看所需镜像
kubeadm config images list
# 所需镜像如下：
k8s.gcr.io/kube-apiserver:v1.22.11
k8s.gcr.io/kube-controller-manager:v1.22.11
k8s.gcr.io/kube-scheduler:v1.22.11
k8s.gcr.io/kube-proxy:v1.22.11
k8s.gcr.io/pause:3.5
k8s.gcr.io/etcd:3.5.0-0
k8s.gcr.io/coredns/coredns:v1.8.4

# 下载镜像(国内镜像使用阿里云地址)
docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/kube-apiserver:v1.22.11
docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/kube-controller-manager:v1.22.11
docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/kube-scheduler:v1.22.11
docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/kube-proxy:v1.22.11
docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/pause:3.5
docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/etcd:3.5.0-0
docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/coredns:v1.8.4

初始化主节点

#所有机器添加master域名映射，以下IP地址需要修改为自己的
echo "172.31.0.3 cluster-endpoint" >> /etc/hosts # master节点 每个节点都需要执行，让每个节点知道master节点

#主节点初始化 只需要在master节点运行
kubeadm init \
--apiserver-advertise-address=172.31.0.3 \ # 填写master节点ip
--control-plane-endpoint=cluster-endpoint \ # 填写设置的域名
--image-repository registry.cn-hangzhou.aliyuncs.com/google_containers \ # 填写使用的镜像地址
--kubernetes-version v1.22.11 \ # 这里的版本要对应上
--service-cidr=10.96.0.0/16 \
--pod-network-cidr=192.168.0.0/16
# 要是想改service-cidr、pod-network-cidr，则所有网络范围需不重叠

• 问题1：(kubernetes v1.24.2)执行kubeadm init命令会出现如下报错信息：

[init] Using Kubernetes version: v1.24.2
[preflight] Running pre-flight checks
        [WARNING Hostname]: hostname "k8s-master" could not be reached
        [WARNING Hostname]: hostname "k8s-master": lookup k8s-master on 114.114.114.114:53: no such host
error execution phase preflight: [preflight] Some fatal errors occurred:
        [ERROR CRI]: container runtime is not running: output: E0710 17:25:48.048814    7677 remote_runtime.go:925] "Status from runtime service failed" err="rpc error: code = Unimplemented desc = unknown service runtime.v1alpha2.RuntimeService"
time="2022-07-10T17:25:48+08:00" level=fatal msg="getting status of runtime: rpc error: code = Unimplemented desc = unknown service runtime.v1alpha2.RuntimeService"
, error: exit status 1
[preflight] If you know what you are doing, you can make a check non-fatal with `--ignore-preflight-errors=...`
To see the stack trace of this error execute with --v=5 or higher

解决方法(参考Kubeadm unknown service runtime.v1alpha2.RuntimeService · Issue #4581 · containerd/containerd · GitHub)：

rm -rf /etc/containerd/config.toml
systemctl restart containerd
kubeadm reset
# 再重新执行kubeadm init即可

• 问题2：(kubernetes v1.24.2)解决问题1后执行kubeadm init命令会出现如下报错信息：

Unfortunately, an error has occurred:
        timed out waiting for the condition

This error is likely caused by:
        - The kubelet is not running
        - The kubelet is unhealthy due to a misconfiguration of the node in some way (required cgroups disabled)

If you are on a systemd-powered system, you can try to troubleshoot the error with the following commands:
        - 'systemctl status kubelet'
        - 'journalctl -xeu kubelet'

Additionally, a control plane component may have crashed or exited when started by the container runtime.
To troubleshoot, list all containers using your preferred container runtimes CLI.
Here is one example how you may list all running Kubernetes containers by using crictl:
        - 'crictl --runtime-endpoint unix:///var/run/containerd/containerd.sock ps -a | grep kube | grep -v pause'
        Once you have found the failing container, you can inspect its logs with:
        - 'crictl --runtime-endpoint unix:///var/run/containerd/containerd.sock logs CONTAINERID'
error execution phase wait-control-plane: couldn't initialize a Kubernetes cluster
To see the stack trace of this error execute with --v=5 or higher

执行journalctl -xeu kubelet | grep Failed命令查看错误，发现还是会去拉取k8s.gcr.io/pause镜像，外网被墙是访问不到的。

解决方法：由于安装的版本是1.24.x，Docker Shim在该版本就被移除了，也就是无法使用Docker，提前拉取下来的镜像没有用，还是会用另外的容器工具拉取，换成低版本如1.22.x版本即可

# 先卸载三大件，再重新安装
yum -y remove kubelet kubeadm kubectl

初始化成功结果：

Your Kubernetes control-plane has initialized successfully!

To start using your cluster, you need to run the following as a regular user:

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

Alternatively, if you are the root user, you can run:

  export KUBECONFIG=/etc/kubernetes/admin.conf

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  https://kubernetes.io/docs/concepts/cluster-administration/addons/

You can now join any number of control-plane nodes by copying certificate authorities
and service account keys on each node and then running the following as root:

  kubeadm join cluster-endpoint:6443 --token o9dn99.eog5utmmykf94zid \
        --discovery-token-ca-cert-hash sha256:73355ba74eb0c362e111b5b6bc910c662fe73dbb29067c5ce055bbbeb66da9ed \
        --control-plane

Then you can join any number of worker nodes by running the following on each as root:

kubeadm join cluster-endpoint:6443 --token o9dn99.eog5utmmykf94zid \
        --discovery-token-ca-cert-hash sha256:73355ba74eb0c362e111b5b6bc910c662fe73dbb29067c5ce055bbbeb66da9ed

设置.kube/config

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

部署网络插件

curl https://docs.projectcalico.org/manifests/calico.yaml -O
kubectl apply -f calico.yaml

加入node节点

# 以下命令24小时有效
kubeadm join cluster-endpoint:6443 --token o9dn99.eog5utmmykf94zid \
        --discovery-token-ca-cert-hash sha256:73355ba74eb0c362e111b5b6bc910c662fe73dbb29067c5ce055bbbeb66da9ed

构建成功

使用kubectl get nodes命令查看是否成功，若节点状态都为Ready则表示成功。