NFS storage

This section covers the setup and use of NFS volumes from a kubernetes cluster. Firstly an nfs server is set up, tested from a client, and then configured for ue in a kubernetes manifest.

NFS Server

To prepare the NFS server for use, using the exported directory /Data01/nfs as an example:

  • apt install nfs-kernel-server
  • mkdir /Data01/nfs
  • chmod go+w /Data01/nfs
  • chown -R nobody:nogroup /Data01/nfs
  • Add entry to /etc/exports: /Data01/nfs *(rw,sync,no_subtree_check)
  • exportfs -a

Client test (OS level)

To test the NFS server, set up a client machine similar to a kubernetes node (assuming the dataserver has IP address 192.168.1.26):

  • Install client driver: apt install nfs-common
  • /etc/fstab: 192.168.1.26:/Data01/nfs /mnt/Data01-nfs nfs rsize=8192,wsize=8192,timeo=14,intr
  • Mount all: mount -a
  • To examine the mounted volume, use df -H, which for the setup above would be something like: *192.168.1.26:/Data01/nfs 2.1T 118G 1.8T 7% /mnt/Data01-nfs*

Kubernetes client

Dynamic Provisioning

Static provisioning is discussed below, but I could only get success with dynamic provisioning.

Dynamic Provisioning for NFS only requires the nfs-common package on k8s nodes: sudo apt install nfs-common -y

The following manifest snippet illustrates usage:

apiVersion: apps/v1
kind: Deployment
...
spec:
...
  template:
...
    spec:
...
      containers:
        volumeMounts:
        - mountPath: /web
          name: radiovolume
          subPath: icecastweb
      initContainers:
...
        volumeMounts:
        - mountPath: /mnt
          name: radiovolume
          subPath: icecastweb
      volumes:
      - name: radiovolume
        nfs:
          path: /Data02/musoclub
          server: 192.168.1.26

CSI Driver for static provisioning

An NFS CSI driver was first attempted, but application had showstopper issues.

  • k8s nodes: use ansible to sudo apt install nfs-common
  • k8s cluster: use flux to install csi-driver-nfs helm chart
  • helm3 repo add csi-driver-nfs https://raw.githubusercontent.com/kubernetes-csi/csi-driver-nfs/master/charts
  • microk8s helm3 repo update
  • helm3 install csi-driver-nfs csi-driver-nfs/csi-driver-nfs --namespace kube-system --set kubeletDir=/var/snap/microk8s/common/var/lib/kubelet
  • chmod -R 777 /srv/nfs
  • chown -R nobody:nogroup /srv/nfs
  • Add volumes into /etc/exports, e.g /srv/nfs 192.168.0.0/24(rw,all_squash,sync,no_subtree_check)

flux install

Issue

  • Pod describe reports: Warning FailedMount 3m32s (x448 over 14h) kubelet MountVolume.SetUp failed for volume "pvc-1c0b7728-5ebd-4bb0-85ba-0f0877ab9080" : rpc error: code = Internal desc = mkdir /var/snap: read-only file system
  • Attempting dynamic provisioning...

References