1. 程序员快看-教程，程序员编程资料站
  2. 首页
  3. K8s实战（2）
  4. 正文

27、K8s实战：Kubernetes安装Redis集群的两个方案

分类：K8s实战（2）    2024-03-10    阅读(124)

一、自定义 yaml 文件安装

背景

在Kubernetes 中部署 Redis 集群面临挑战，因为每个 Redis 实例都依赖于一个配置文件，该文件可以跟踪其他集群实例及其角色。为此，需要结合使用 Kubernetes StatefulSets 和 PersistentVolumes 实现。Redis 集群架构如下图所示：

*

创建 StatefulSet yaml 文件

---
apiVersion:*v1
kind:*ConfigMap
metadata:
**name:*redis-cluster
data:
**update.sh:*|
****#!/bin/sh
****REDIS_NODES="/data/nodes.conf"
****sed*-i*-e*"/myself/*s/[0-9]\{1,3\}\.[0-9]\{1,3\}\.[0-9]\{1,3\}\.[0-9]\{1,3\}/${POD_IP}/"*${REDIS_NODES}
****exec*"$@"
**redis.conf:*|+
****bind*0.0.0.0
****cluster-enabled*yes
****cluster-require-full-coverage*no
****cluster-node-timeout*30000
****cluster-config-file*/data/nodes.conf
****cluster-migration-barrier*1
****appendonly*yes
****protected-mode*no
---
apiVersion:*apps.kruise.io/v1beta1
#*apiVersion:*apps/v1
kind:*StatefulSet
metadata:
**name:*redis-cluster
spec:
**serviceName:*redis-cluster
**replicas:*6
**selector:
****matchLabels:
******app:*redis-cluster
**template:
****metadata:
******labels:
********app:*redis-cluster
****spec:
******containers:
******-*name:*redis
********image:*redis:6.2.1-alpine
********ports:
********-*containerPort:*6379
**********name:*client
********-*containerPort:*16379
**********name:*gossip
********command:*["/conf/update.sh",*"redis-server",*"/conf/redis.conf"]
********env:
********-*name:*POD_IP
**********valueFrom:
************fieldRef:
**************fieldPath:*status.podIP
********volumeMounts:
********-*name:*conf
**********mountPath:*/conf
**********readOnly:*false
********-*name:*data
**********mountPath:*/data
**********readOnly:*false
******volumes:
******-*name:*conf
********configMap:
**********name:*redis-cluster
**********defaultMode:*0755
**volumeClaimTemplates:
**-*metadata:
******name:*data
****spec:
******accessModes:*[*"ReadWriteOnce"*]
******resources:
********requests:
**********storage:*100Gi
******storageClassName:*rbd

解释：

cluster-migration-barrier 那些分配后仍然剩余 migration barrier 个从节点的主节点才会触发节点分配，而不是分配前有 migration barrier 个从节点的主节点就会触发节点分配，默认是 1，生产环境建议维持默认值。
protected-mode no 参数是为了禁止外网访问 redis，如果启用了，则只能够通过 lookback ip（127.0.0.1）访问 Redis，如果外网访问 redis，会报出异常。
apiVersion: apps.kruise.io/v1beta1 控制器这里使用 kruise 提供的 Advanced StatefulSet，如果集群没有安装 kruise，可以使用 apps/v1。

安装 redis 集群

[root@qd01-stop-k8s-master001*redis]#*kubectl*apply*-f*install-redis.yaml
configmap/redis-cluster*created
statefulset.apps.kruise.io/redis-cluster*created

[root@qd01-stop-k8s-master001*redis]#*kubectl*get*po*-n*op
NAME**************READY***STATUS****RESTARTS***AGE
redis-cluster-0***1/1*****Running***0**********3m26s
redis-cluster-1***1/1*****Running***0**********3m14s
redis-cluster-2***1/1*****Running***0**********2m54s
redis-cluster-3***1/1*****Running***0**********2m23s
redis-cluster-4***1/1*****Running***0**********2m14s
redis-cluster-5***1/1*****Running***0**********114s

创建 redis 集群的 service

---
apiVersion:*v1
kind:*Service
metadata:
**name:*redis-cluster
**namespace:*op
spec:
**type:*ClusterIP
**ports:
**-*port:*6379
****targetPort:*6379
****name:*client
**-*port:*16379
****targetPort:*16379
****name:*gossip
**selector:
****app:*redis-cluster

[root@qd01-stop-k8s-master001*redis]#*kubectl*apply*-f*redis-svc.yml
service/redis-cluster*created
[root@qd01-stop-k8s-master001*redis]#*kubectl*get*svc*-n*op
NAME************TYPE********CLUSTER-IP******EXTERNAL-IP***PORT(S)**************AGE
redis-cluster***ClusterIP***10.97.197.224***<none>********6379/TCP,16379/TCP***9s

测试能后连通
[root@qd01-stop-k8s-master001*redis]#*telnet**10.97.197.224*6379
Trying*10.97.197.224...
Connected*to*10.97.197.224.
Escape*character*is*'^]'.

初始化 redis 集群

执行如下命令，获取到 pod IP，然后使用 redis-cli --cluster 创建集群

[root@qd01-stop-k8s-master001*redis]#*kubectl*-n*op*exec*-it*redis-cluster-0*--*redis-cli*--cluster*create*--cluster-replicas*1*$(kubectl*-n*op*get*pods*-l*app=redis-cluster*-o*jsonpath='{range.items[*]}{.status.podIP}:6379*{end}')
>>>*Performing*hash*slots*allocation*on*6*nodes...
Master[0]*->*Slots*0*-*5460
Master[1]*->*Slots*5461*-*10922
Master[2]*->*Slots*10923*-*16383
Adding*replica*100.88.43.67:6379*to*100.64.147.152:6379
Adding*replica*100.113.170.5:6379*to*100.98.174.217:6379
Adding*replica*100.64.147.153:6379*to*100.80.158.227:6379
M:*b47b27a3dbddf3fc1370cbe14ae753f4fce20b04*100.64.147.152:6379
***slots:[0-5460]*(5461*slots)*master
M:*09543217c903350e963fc4fdf4acb73f8a1b7f8b*100.98.174.217:6379
***slots:[5461-10922]*(5462*slots)*master
M:*5389ace495b68eeac85370d6783648dff68f2fb6*100.80.158.227:6379
***slots:[10923-16383]*(5461*slots)*master
S:*b1f39714c006ae55b12b18e6537303d7a00e1704*100.64.147.153:6379
***replicates*5389ace495b68eeac85370d6783648dff68f2fb6
S:*0113f4668ec2f3ca2e9470c44bd5faab532b0936*100.88.43.67:6379
***replicates*b47b27a3dbddf3fc1370cbe14ae753f4fce20b04
S:*e1e2f18ae66c79f1943390beabb59613abbad38a*100.113.170.5:6379
***replicates*09543217c903350e963fc4fdf4acb73f8a1b7f8b
Can*I*set*the*above*configuration?*(type*'yes'*to*accept):*yes
>>>*Nodes*configuration*updated
>>>*Assign*a*different*config*epoch*to*each*node
>>>*Sending*CLUSTER*MEET*messages*to*join*the*cluster
Waiting*for*the*cluster*to*join
..
>>>*Performing*Cluster*Check*(using*node*100.64.147.152:6379)
M:*b47b27a3dbddf3fc1370cbe14ae753f4fce20b04*100.64.147.152:6379
***slots:[0-5460]*(5461*slots)*master
***1*additional*replica(s)
S:*0113f4668ec2f3ca2e9470c44bd5faab532b0936*100.88.43.67:6379
***slots:*(0*slots)*slave
***replicates*b47b27a3dbddf3fc1370cbe14ae753f4fce20b04
M:*09543217c903350e963fc4fdf4acb73f8a1b7f8b*100.98.174.217:6379
***slots:[5461-10922]*(5462*slots)*master
***1*additional*replica(s)
M:*5389ace495b68eeac85370d6783648dff68f2fb6*100.80.158.227:6379
***slots:[10923-16383]*(5461*slots)*master
***1*additional*replica(s)
S:*e1e2f18ae66c79f1943390beabb59613abbad38a*100.113.170.5:6379
***slots:*(0*slots)*slave
***replicates*09543217c903350e963fc4fdf4acb73f8a1b7f8b
S:*b1f39714c006ae55b12b18e6537303d7a00e1704*100.64.147.153:6379
***slots:*(0*slots)*slave
***replicates*5389ace495b68eeac85370d6783648dff68f2fb6
[OK]*All*nodes*agree*about*slots*configuration.
>>>*Check*for*open*slots...
>>>*Check*slots*coverage...
[OK]*All*16384*slots*covered.

验证 Redis 集群

[root@qd01-stop-k8s-master001*redis]#*kubectl*-n*op**exec*-it*redis-cluster-0*--*redis-cli*cluster*info
cluster_state:ok
cluster_slots_assigned:16384
cluster_slots_ok:16384
cluster_slots_pfail:0
cluster_slots_fail:0
cluster_known_nodes:6
cluster_size:3
cluster_current_epoch:6
cluster_my_epoch:1
cluster_stats_messages_ping_sent:178
cluster_stats_messages_pong_sent:181
cluster_stats_messages_sent:359
cluster_stats_messages_ping_received:176
cluster_stats_messages_pong_received:178
cluster_stats_messages_meet_received:5
cluster_stats_messages_received:359

[root@qd01-stop-k8s-master001*redis]#*kubectl*-n*op**exec*-it*redis-cluster-0*--*redis-cli*cluster*nodes
0113f4668ec2f3ca2e9470c44bd5faab532b0936*100.88.43.67:6379@16379*slave*b47b27a3dbddf3fc1370cbe14ae753f4fce20b04*0*1615348311156*1*connected
09543217c903350e963fc4fdf4acb73f8a1b7f8b*100.98.174.217:6379@16379*master*-*0*1615348314162*2*connected*5461-10922
b47b27a3dbddf3fc1370cbe14ae753f4fce20b04*100.64.147.152:6379@16379*myself,master*-*0*1615348312000*1*connected*0-5460
5389ace495b68eeac85370d6783648dff68f2fb6*100.80.158.227:6379@16379*master*-*0*1615348312000*3*connected*10923-16383
e1e2f18ae66c79f1943390beabb59613abbad38a*100.113.170.5:6379@16379*slave*09543217c903350e963fc4fdf4acb73f8a1b7f8b*0*1615348313160*2*connected
b1f39714c006ae55b12b18e6537303d7a00e1704*100.64.147.153:6379@16379*slave*5389ace495b68eeac85370d6783648dff68f2fb6*0*1615348312158*3*connected

从输出可以看到，集群总共 6 个节点，三主三从

二、使用 kubeDB 安装

安装 KubeDB

获取AppsCode License：https://license-issuer.appscode.com/；下载 KubeDB charts：https://github.com/appscode/charts/tree/master/stable/kubedb-community

[root@qd01-stop-k8s-master001*kubedb-community]#*unzip*kubedb-community-v0.16.2.tgz
[root@qd01-stop-k8s-master001*kubedb-community]#*cd*kubedb-community
[root@qd01-stop-k8s-master001*kubedb-community]#*ls*-al
total*96
drwxr-xr-x*4*root*root***158*Mar*10*15:26*.
drwxr-xr-x*3*root*root****66*Mar*10*15:24*..
-rw-r--r--*1*root*root***351*Feb*16*09:55*Chart.yaml
drwxr-xr-x*2*root*root****28*Mar*10*15:24*ci
-rw-r--r--*1*root*root***493*Feb*16*09:55*doc.yaml
-rw-r--r--*1*root*root***353*Feb*16*09:55*.helmignore
-rw-r--r--*1*root*root*24422*Feb*16*09:55*README.md
drwxr-xr-x*2*root*root**4096*Mar*10*15:24*templates
-rw-r--r--*1*root*root*47437*Feb*16*09:55*values.openapiv3_schema.yaml
-rw-r--r--*1*root*root**5230*Feb*16*09:55*values.yaml

修改values.yaml，把 License 文件放到 kubedb-community 目录下

使用 helm 安装

[root@qd01-stop-k8s-master001*kubedb-community]#*helm*install*kubedb-community*--namespace*kube-system*--set-file*license=./kubedb-community-license.txt*-f*values.yaml**.
NAME:*kubedb-community
LAST*DEPLOYED:*Wed*Mar*10*15:38:59*2021
NAMESPACE:*kube-system
STATUS:*deployed
REVISION:*1
TEST*SUITE:*None
NOTES:
To*verify*that*KubeDB*has*started,*run:
**kubectl*get*deployment*--namespace*kube-system*-l*"app.kubernetes.io/name=kubedb-community,app.kubernetes.io/instance=kubedb-community"
Now*install/upgrade*appscode/kubedb-catalog*chart.
To*install,*run:
**helm*install*kubedb-catalog*appscode/kubedb-catalog*--version*v0.16.2*--namespace*kube-system
To*upgrade,*run:
**helm*upgrade*kubedb-catalog*appscode/kubedb-catalog*--version*v0.16.2*--namespace*kube-system

执行如下命令查看是否安装完成
[root@qd01-stop-k8s-master001*kubedb-community]#*kubectl*get*deployment*--namespace*kube-system*-l*"app.kubernetes.io/name=kubedb-community,app.kubernetes.io/instance=kubedb-community"
NAME***************READY***UP-TO-DATE***AVAILABLE***AGE
kubedb-community***1/1*****1************1***********38s

等待crds注册成功
[root@qd01-stop-k8s-master001*kubedb-community]#*kubectl*get*crds*-l*app.kubernetes.io/name=kubedb*-w
NAME***************************************CREATED*AT
elasticsearches.kubedb.com*****************2021-03-10T07:39:42Z
elasticsearchversions.catalog.kubedb.com***2021-03-10T07:39:45Z
etcds.kubedb.com***************************2021-03-10T07:39:42Z
etcdversions.catalog.kubedb.com************2021-03-10T07:39:45Z
memcacheds.kubedb.com**********************2021-03-10T07:39:43Z
memcachedversions.catalog.kubedb.com*******2021-03-10T07:39:45Z
mongodbs.kubedb.com************************2021-03-10T07:39:43Z
mongodbversions.catalog.kubedb.com*********2021-03-10T07:39:45Z
mysqls.kubedb.com**************************2021-03-10T07:39:43Z
mysqlversions.catalog.kubedb.com***********2021-03-10T07:39:46Z
perconaxtradbs.kubedb.com******************2021-03-10T07:39:43Z
perconaxtradbversions.catalog.kubedb.com***2021-03-10T07:39:46Z
pgbouncers.kubedb.com**********************2021-03-10T07:39:44Z
pgbouncerversions.catalog.kubedb.com*******2021-03-10T07:39:46Z
postgreses.kubedb.com**********************2021-03-10T07:39:44Z
postgresversions.catalog.kubedb.com********2021-03-10T07:39:46Z
proxysqls.kubedb.com***********************2021-03-10T07:39:44Z
proxysqlversions.catalog.kubedb.com********2021-03-10T07:39:46Z
redises.kubedb.com*************************2021-03-10T07:39:45Z
redisversions.catalog.kubedb.com***********2021-03-10T07:39:46Z

安装 KubeDB Catalog

先下载：https://github.com/appscode/charts/tree/master/stable/kubedb-catalog

[root@qd01-stop-k8s-master001*kubedb-catalog]#*tar*-zxf*kubedb-catalog-v0.16.2.tgz
[root@qd01-stop-k8s-master001*kubedb-catalog]#*cd*kubedb-catalog
[root@qd01-stop-k8s-master001*kubedb-catalog]#*ls*-al
total*24
drwxr-xr-x**3*root*root**148*Mar*10*15:48*.
drwxr-xr-x**3*root*root***28*Mar*10*15:48*..
-rw-r--r--**1*root*root**321*Jan*26*20:08*Chart.yaml
-rw-r--r--**1*root*root**467*Jan*26*20:08*doc.yaml
-rw-r--r--**1*root*root**353*Jan*26*20:08*.helmignore
-rw-r--r--**1*root*root*3195*Jan*26*20:08*README.md
drwxr-xr-x*12*root*root**188*Mar*10*15:48*templates
-rw-r--r--**1*root*root**744*Jan*26*20:08*values.openapiv3_schema.yaml
-rw-r--r--**1*root*root*1070*Jan*26*20:08*values.yaml

[root@qd01-stop-k8s-master001*kubedb-catalog]#*helm*install*kubedb-catalog*--namespace*kube-system*-f*values.yaml**.
NAME:*kubedb-catalog
LAST*DEPLOYED:*Wed*Mar*10*15:50:50*2021
NAMESPACE:*kube-system
STATUS:*deployed
REVISION:*1
TEST*SUITE:*None

使用 kubedb 安装 Redis 集群

官方给的 Redis 生命周期如下图所示：

*

kubedb 安装 redis 支持如下特性：

Features****Availability
Clustering*****
Instant*Backup*****
Scheduled*Backup*****
Persistent*Volume*****
Initialize*using*Snapshot*****
Initialize*using*Script*****
Custom*Configuration*****
Using*Custom*docker*image*****
Builtin*Prometheus*Discovery*****
Using*Prometheus*operator*****

查看支持的版本

[root@qd01-stop-k8s-master001*kubedb-catalog]#*kubectl*get*redisversions
NAME*******VERSION***DB_IMAGE****************DEPRECATED***AGE
4.0.11*****4.0.11****kubedb/redis:4.0.11******************15m
4.0.6-v2***4.0.6*****kubedb/redis:4.0.6-v2****************15m
5.0.3-v1***5.0.3*****kubedb/redis:5.0.3-v1****************15m
6.0.6******6.0.6*****kubedb/redis:6.0.6*******************15m

编辑 yaml 安装文件

可以参照：https://github.com/kubedb/docs/blob/v2021.01.26/docs/examples/redis/clustering/demo-1.yaml。这里选择安装 6.0.6这个版本，我的集群 storageClassName: "rbd"，请根据实际修改。如果想自定义redis.conf，请参考：https://github.com/kubedb/docs/blob/v2021.01.26/docs/examples/redis/custom-config/redis-custom.yaml。

apiVersion:*kubedb.com/v1alpha2
kind:*Redis
metadata:
**name:*redis-cluster
**namespace:*op
spec:
**version:*6.0.6
**mode:*Cluster
**cluster:
****master:*3
****replicas:*1
**storageType:*Durable
**storage:
****resources:
******requests:
********storage:*1Gi
****storageClassName:*"rbd"
****accessModes:
******-*ReadWriteOnce

执行安装

[root@qd01-stop-k8s-master001*kubedb-community]#*kubectl*apply*-f*redis-cluster.yaml
redis.kubedb.com/redis-cluster*created

安装完成，可以如下查看
[root@qd01-stop-k8s-master001*kubedb-community]#*kubectl*get*rd,po*-n*op
NAME*****************************VERSION***STATUS*********AGE
redis.kubedb.com/redis-cluster***6.0.6*****Provisioning***6m55s

NAME*************************READY***STATUS****RESTARTS***AGE
pod/redis-cluster-shard0-0***1/1*****Running***0**********6m54s
pod/redis-cluster-shard0-1***1/1*****Running***0**********6m18s
pod/redis-cluster-shard1-0***1/1*****Running***0**********5m38s
pod/redis-cluster-shard1-1***1/1*****Running***0**********5m1s
pod/redis-cluster-shard2-0***1/1*****Running***0**********4m30s
pod/redis-cluster-shard2-1***1/1*****Running***0**********4m8s

[root@qd01-stop-k8s-master001*redis]#*kubectl*get*svc*-n*op
NAME*****************TYPE********CLUSTER-IP******EXTERNAL-IP***PORT(S)****AGE
redis-cluster********ClusterIP***10.97.197.224***<none>********6379/TCP***5h16m
redis-cluster-pods***ClusterIP***None************<none>********6379/TCP***17m

验证集群

[root@qd01-stop-k8s-master001*kubedb-community]#*kubectl*get*pods*-n**op**-o*jsonpath='{range.items[*]}{.metadata.name}*----------*{.status.podIP}:6379{"\t\n"}{end}'*|*grep*redis
redis-cluster-shard0-0*----------*100.64.147.156:6379
redis-cluster-shard0-1*----------*100.98.174.218:6379
redis-cluster-shard1-0*----------*100.126.252.204:6379
redis-cluster-shard1-1*----------*100.113.170.6:6379
redis-cluster-shard2-0*----------*100.107.55.69:6379
redis-cluster-shard2-1*----------*100.78.230.4:6379

[root@qd01-stop-k8s-master001*redis]#*kubectl*-n*op**exec*-it*redis-cluster-shard0-0**--*redis-cli*cluster*info
cluster_state:ok
cluster_slots_assigned:16384
cluster_slots_ok:16384
cluster_slots_pfail:0
cluster_slots_fail:0
cluster_known_nodes:6
cluster_size:3
cluster_current_epoch:3
cluster_my_epoch:1
cluster_stats_messages_ping_sent:864
cluster_stats_messages_pong_sent:882
cluster_stats_messages_sent:1746
cluster_stats_messages_ping_received:879
cluster_stats_messages_pong_received:864
cluster_stats_messages_meet_received:3
cluster_stats_messages_received:1746

[root@qd01-stop-k8s-master001*redis]#*kubectl*-n*op**exec*-it*redis-cluster-shard0-0**--*redis-cli*cluster*nodes
1895cb4b9c31b848666c61000e502f55a29a8255*100.64.147.155:6379@16379*master*-*0*1615365162008*2*connected*5461-10922
30bdbf2ca37001774498a9b935afbc1cd2ce389c*100.126.252.203:6379@16379*slave*2c06092fafa99e0158e39e6237a04fed25be3550*0*1615365163000*1*connected
9b2cfbd5c1b417121d410141b6da9512ad29ce3c*100.78.230.3:6379@16379*slave*e83446c368839c5fdccf5f70e3b1004eb67cb651*0*1615365163512*3*connected
2c06092fafa99e0158e39e6237a04fed25be3550*100.82.197.130:6379@16379*myself,master*-*0*1615365162000*1*connected*0-5460
1379d2b20f26ab13d53068d276ec5d988b7a0273*100.64.122.197:6379@16379*slave*1895cb4b9c31b848666c61000e502f55a29a8255*0*1615365163000*2*connected
e83446c368839c5fdccf5f70e3b1004eb67cb651*100.107.55.68:6379@16379*master*-*0*1615365164014*3*connected*10923-16383

版权声明：本文不是「本站」原创文章，版权归原作者所有 | 原文地址: