Kubernetes默认提供了非常丰富的Volume类型，代码都在这里。

环境准备

Kubernetes集群,关于集群搭建可以参考下我的另一篇文章：kubeadm搭建Kubernetes 1.15高可用集群 。

1
2
3
4
5
6
7
8

➜  ~ kubectl get nodes
NAME      STATUS   ROLES    AGE   VERSION
kadm01    Ready    master   9d    v1.15.5
kadm02    Ready    master   9d    v1.15.5
kadm03    Ready    master   9d    v1.15.5
knode01   Ready    node     9d    v1.15.5
knode02   Ready    node     9d    v1.15.5
knode03   Ready    node     9d    v1.15.5

搭建NFS Server

这里为了简单就直接在kubernetes集群里以StatefulSet的方式起一个NFS Server。

1

➜  cat nfs-sts.yaml

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48

apiVersion: v1
kind: Service
metadata:
  name: nfs
  labels:
    app: nfs
spec:
  ports:
  - port: 2049
    name: tcp
  clusterIP: None
  selector:
    app: nfs
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: nfs
spec:
  selector:
    matchLabels:
      app: nfs
  serviceName: "nfs"
  replicas: 1
  template:
    metadata:
      labels:
        app: nfs
    spec:
      containers:
      - name: nfs
        image: itsthenetwork/nfs-server-alpine:12
        env:
        - name: SHARED_DIRECTORY
          value: "/nfsshare"
        securityContext:
          capabilities:
            add: ["SYS_ADMIN", "SETPCAP"]
        ports:
        - containerPort: 2049
          name: tcp
        volumeMounts:
        - name: nfs-data
          mountPath: /nfsshare 
      volumes:
      - name: nfs-data
        hostPath:
          path: /data/nfs

验证服务启动状态：

1
2
3
4
5
6

➜  k8s kubectl get all -l app=nfs
NAME        READY   STATUS    RESTARTS   AGE
pod/nfs-0   1/1     Running   0          44h

NAME          TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)    AGE
service/nfs   ClusterIP   None         <none>        2049/TCP   44h

创建PV、PVC

1

➜  cat nfs-pvc.yaml

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28

apiVersion: v1
kind: PersistentVolume
metadata:
  name: nfs
  labels:
    pv: nfs-pv
spec:
  capacity:
    storage: 10Mi
  accessModes:
    - ReadWriteMany
  nfs:
    server: "10.1.25.130"
    path: "/"
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: nfs
spec:
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 10Mi
  selector:
    matchLabels:
      pv: nfs-pv
  • 这里需要注意一下：server地址不能写集群的dns地址：nfs.default.svc.cluster.local，因为需要先mount本地，再映射到pod中。
  • The storage parameter is used for matching PVC to a PV, and for autoprovisioning PVs when supported。所以这里指定的storage: 10Mi对NFS是无效的，因为它没有quota的概念。

验证:

1
2
3
4
5
6

➜  k8s kubectl get pv,pvc
NAME                   CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM         STORAGECLASS   REASON   AGE
persistentvolume/nfs   10Mi       RWX            Retain           Bound    default/nfs                           45h

NAME                        STATUS   VOLUME   CAPACITY   ACCESS MODES   STORAGECLASS   AGE
persistentvolumeclaim/nfs   Bound    nfs      10Mi       RWX                           45h

挂载PVC

1

➜ cat nginx-test.yaml

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-test-pvc
spec:
  selector:
    matchLabels:
      run: nginx-test-pvc
  replicas: 1
  template:
    metadata:
      labels:
        run: nginx-test-pvc
    spec:
      containers:
      - name: nginx-test-pvc
        image: nginx:1.17
        volumeMounts:
        - mountPath: /usr/share/nginx/html
          name: nginx-data
        ports:
        - containerPort: 80
      volumes:
      - name: nginx-data
        persistentVolumeClaim:
          claimName: nfs

验证

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12

➜ kubectl exec nginx-test-pvc-cdf6b9597-blr9x df
Filesystem                1K-blocks    Used Available Use% Mounted on
overlay                    83329276 3926636  79402640   5% /
tmpfs                         65536       0     65536   0% /dev
tmpfs                       8199208       0   8199208   0% /sys/fs/cgroup
/dev/mapper/vg_001-lv_001  83329276 3926636  79402640   5% /etc/hosts
shm                           65536       0     65536   0% /dev/shm
10.1.25.130:/              83330048 3927040  79403008   5% /usr/share/nginx/html
tmpfs                       8199208      12   8199196   1% /run/secrets/kubernetes.io/serviceaccount
tmpfs                       8199208       0   8199208   0% /proc/acpi
tmpfs                       8199208       0   8199208   0% /proc/scsi
tmpfs                       8199208       0   8199208   0% /sys/firmware

调试

  • Kubernetes node 节点需要安装 nfs-utils 包，不然为报下边错误：
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

Mounting command: systemd-run
Mounting arguments: --description=Kubernetes transient mount for /var/lib/kubelet/pods/0b0b2952-e61f-4f96-a110-414637793a81/volumes/kubernetes.io~nfs/nfs --scope -- mount -t nfs nfs.default.svc.cluster.local:/ /var/lib/kubelet/pods/0b0b2952-e61f-4f96-a110-414637793a81/volumes/kubernetes.io~nfs/nfs
Output: Running scope as unit run-28692.scope.
mount: wrong fs type, bad option, bad superblock on nfs.default.svc.cluster.local:/,
       missing codepage or helper program, or other error
       (for several filesystems (e.g. nfs, cifs) you might
       need a /sbin/mount.<type> helper program)

       In some cases useful info is found in syslog - try
       dmesg | tail or so.
  Warning  FailedMount  96s  kubelet, knode03  MountVolume.SetUp failed for volume "nfs" : mount failed: exit status 32

参考

https://kubernetes.io/zh/docs/concepts/storage/volumes/

https://github.com/kubernetes/kubernetes/blob/master/pkg/volume/nfs/nfs.go