机器准备

6台虚拟机
IP地址：

10.1.1.10~12

10.1.1.20~22

内核：4.4.169-1.el7.elrepo.x86_64（3.10以上能装docker都可）

基本软件安装

keepalived安装

略

ssh实现集群机器间互相访问

略

docker 安装

本文用的docker版本：19.03.0-ce
过程：略
安装成功后继续下一步

基本配置

进入 step2-conf 目录下，执行 basic_conf.sh（脚本代码如下，加了注释）：

#!/bin/sh

# 配置docker开机启动
systemctl daemon-reload
systemctl restart docker
systemctl enable docker

# 关闭防火墙并关闭开机启动
systemctl disable firewalld
systemctl stop firewalld

# 永久关闭 selinux
setenforce 0
sed "s/^SELINUX=enforcing$/SELINUX=disabled/g" /etc/selinux/config

# 关闭swap
# 网络设置
swapoff -a
yes|cp /etc/fstab /etc/fstab.bak
cat /etc/fstab.bak |grep -v swap > /etc/fstab

echo '''
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_forward = 1
vm.swappiness=0''' > /etc/sysctl.d/k8s.conf

modprobe br_netfilter
sysctl -p /etc/sysctl.d/k8s.conf

# 配置 ipvs的依赖模块
cat > /etc/sysconfig/modules/ipvs.modules <<EOF
#!/bin/bash
modprobe -- ip_vs
modprobe -- ip_vs_rr
modprobe -- ip_vs_wrr
modprobe -- ip_vs_sh
modprobe -- nf_conntrack_ipv4
EOF
chmod 755 /etc/sysconfig/modules/ipvs.modules && bash /etc/sysconfig/modules/ipvs.modules && lsmod | grep -e ip_vs -e nf_conntrack_ipv4

# 安装ipset 和ipvsadm
yum install -y ipset
yum install -y ipvsadm

导入系统镜像

k8s集群master需要镜像：kube-apiserver kube-controller-manager kube-scheduler kube-proxy pause，根据安装k8s版本选取对应版本的镜像
k8s 组件安装：
master节点组件： kubeadm、kubectl、kubelet、
node节点组件： kubelet

配置开机启动kubelet

1 2	systemctl start kubelet systemctl enable kubelet

安装外部etcd集群（不使用kubeadm自动生成的etcd）

安装cfslsl工具

wget -O /usr/local/bin/cfssl https://pkg.cfssl.org/R1.2/cfssl_linux-amd64
wget -O /usr/local/bin/cfssljson https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64
wget -O /usr/local/bin/cfssl-certinfo  https://pkg.cfssl.org/R1.2/cfssl-certinfo_linux-amd64
for cfssl in `ls /usr/local/bin/cfssl*`;do chmod +x $cfssl;done;

生成etcd证书：

根据实际情况修改shell中的hosts

#/bin/sh
# gen-cert.sh

mkdir -p /opt/etcd/{ssl,bin,cfg}
cd /opt/etcd/ssl
# ca config
cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=www server-csr.json | cfssljson -bare server
cat > ca-config.json <<EOF
{
  "signing": {
    "default": {
      "expiry": "87600h"
    },
    "profiles": {
      "www": {
         "expiry": "87600h",
         "usages": [
            "signing",
            "key encipherment",
            "server auth",
            "client auth"
        ]
      }
    }
  }
}
EOF

# ca csr config
cat > ca-csr.json <<EOF
{
    "CN": "etcd CA",
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "names": [
        {
            "C": "CN",
            "L": "SZ",
            "ST": "SZ"
        }
    ]
}
EOF

# gen
# 根据实际情况修改hosts
cfssl gencert -initca ca-csr.json | cfssljson -bare ca -
cat > server-csr.json <<EOF
{
    "CN": "etcd",
    "hosts": [
      "10.1.1.10",
      "10.1.1.20",
      "10.1.1.21"
    ],
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "names": [
        {
            "C": "CN",
            "L": "SZ",
            "ST": "SZ"
        }
    ]
}
EOF

cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=www server-csr.json | cfssljson -bare server

# ca.csr            #ca证书签名请求
# ca.pem            #ca证书（公钥）
# ca-key.pem      #ca私钥

# server.csr        #server证书签名请求
# server.pem        #server证书（公钥）
# server-key.pem    #server私钥

安装etcd

节点1：

从github上下载etcd的release版本，解压后进入目录，将etcd二进制文件复制到 /opt/etcd/bin 目录下

1 2	#!/bin/sh cp bin/etcd bin/etcdctl /opt/etcd/bin

安装脚本 install-etcd.sh：

#!/bin/bash
# example: ./etcd.sh etcd01 IP1 etcd02=https://IP2:2380,etcd03=https://IP3:2380

ETCD_NAME=$1
ETCD_IP=$2
ETCD_CLUSTER=$3

WORK_DIR=/opt/etcd

# 构建etcd配置文件
cat <<EOF >$WORK_DIR/cfg/etcd
#[Member]
ETCD_NAME="${ETCD_NAME}"
ETCD_DATA_DIR="/var/lib/etcd/default.etcd" # 
ETCD_LISTEN_PEER_URLS="https://${ETCD_IP}:2380"
ETCD_LISTEN_CLIENT_URLS="https://${ETCD_IP}:2379"

#[Clustering]
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://${ETCD_IP}:2380"
ETCD_ADVERTISE_CLIENT_URLS="https://${ETCD_IP}:2379"
ETCD_INITIAL_CLUSTER="etcd01=https://${ETCD_IP}:2380,${ETCD_CLUSTER}"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_INITIAL_CLUSTER_STATE="new"
EOF

# 构建systemctl服务管理etcd
cat <<EOF >/usr/lib/systemd/system/etcd.service
[Unit]
Description=Etcd Server
After=network.target
After=network-online.target
Wants=network-online.target

[Service]
Type=notify
EnvironmentFile=${WORK_DIR}/cfg/etcd
ExecStart=${WORK_DIR}/bin/etcd \
--name=\${ETCD_NAME} \
--data-dir=\${ETCD_DATA_DIR} \
--listen-peer-urls=\${ETCD_LISTEN_PEER_URLS} \
--listen-client-urls=\${ETCD_LISTEN_CLIENT_URLS},http://127.0.0.1:2379 \
--advertise-client-urls=\${ETCD_ADVERTISE_CLIENT_URLS} \
--initial-advertise-peer-urls=\${ETCD_INITIAL_ADVERTISE_PEER_URLS} \
--initial-cluster=\${ETCD_INITIAL_CLUSTER} \
--initial-cluster-token=\${ETCD_INITIAL_CLUSTER_TOKEN} \
--initial-cluster-state=new \
--cert-file=${WORK_DIR}/ssl/server.pem \
--key-file=${WORK_DIR}/ssl/server-key.pem \
--peer-cert-file=${WORK_DIR}/ssl/server.pem \
--peer-key-file=${WORK_DIR}/ssl/server-key.pem \
--trusted-ca-file=${WORK_DIR}/ssl/ca.pem \
--peer-trusted-ca-file=${WORK_DIR}/ssl/ca.pem
Restart=on-failure
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target
EOF

systemctl daemon-reload
systemctl enable etcd
systemctl restart etcd

此处我使用的IP是 10.1.1.10-12 ，所以执行命令就是 ./install-etcd.sh etcd01 10.1.1.10 etcd02=https://10.1.1.11:2380,etcd03=10.1.1.12:2380

节点2、3

将etcd执行文件及配置复制到其他两个节点：

scp -r /opt/etcd/  root@10.1.1.11:/opt/etcd/
scp -r /opt/etcd/  root@10.1.1.12:/opt/etcd/
scp /usr/lib/systemd/system/etcd.service root@10.1.1.11:/usr/lib/systemd/system/
scp /usr/lib/systemd/system/etcd.service root@10.1.1.12:/usr/lib/systemd/system/

修改配置：

修改/opt/etcd/cfg/etcd 配置文件，MEMBER配置块内容改为该节点自身的配置

Clustering配置块中，ETCD_INITIAL_ADVERTISE_PEER_URLS ETCD_ADVERTISE_CLIENT_URLS 的IP改成当前机器的IP

节点状态查看

1	ETCDCTL_API=3 etcdctl --cacert=/opt/etcd/ssl/ca.pem --cert=/opt/etcd/ssl/server.pem --key=/opt/etcd/ssl/server-key.pem --endpoints=https://10.1.1.10:2379 member list

结果显示：
66a8527f13b8fecc, started, etcd03, https://10.1.1.12:2380, https://10.1.1.12:2379
a6a292fe93a0085d, started, etcd01, https://10.1.1.10:2380, https://10.1.1.10:2379
da873601634f4a11, started, etcd02, https://10.1.1.11:2380, https://10.1.1.11:2379

初始化集群

进入 /etc/kubernetes 目录，vim kubeadm-config.yaml

apiVersion: kubeadm.k8s.io/v1beta2
kind: ClusterConfiguration
kubernetesVersion: v1.16.2
controlPlaneEndpoint: "10.1.1.10:6443"
apiServer:
  certSANs:
  - 10.1.1.10
  - 10.1.1.11
  - 10.1.1.12
  - k8s.local
etcd:
  external:
    endpoints:
    - https://10.1.1.10:2379
    - https://10.1.1.11:2379
    - https://10.1.1.12:2379
    caFile: /opt/etcd/ssl/ca.pem
    certFile: /opt/etcd/ssl/server.pem
    keyFile: /opt/etcd/ssl/server-key.pem 
---
apiVersion: kubeproxy.config.k8s.io/v1alpha1
kind: KubeProxyConfiguration
mode: ipvs

执行初始化命令：

1	kubeadm init --config /etc/kubernetes/kubeadm-config.yaml --upload-certs

命令执行完会生成两个带token 的join命令

kubeadm join xxxx –control-plane xxxx

kubeadm join xxxx
分别用于master节点、node节点加入集群。当加入master节点时如果报证书错误，把执行init节点的 /etc/kubernetes/pki 文件夹复制到待加入的master节点。

其他：

初始化集群失败后，清理etcd数据：

1	ETCDCTL_API=3 etcdctl --cacert=/opt/etcd/ssl/ca.pem --cert=/opt/etcd/ssl/server.pem --key=/opt/etcd/ssl/server-key.pem --endpoints=https://10.1.1.10:2379 del / --prefix

参考：

[1]. etcd集群安装