Weka CSI Plugin

Overview
The Container Storage Interface (CSI) is a standard for exposing arbitrary block and file storage systems to containerized workloads on Container Orchestration Systems (COs) like Kubernetes.
The Weka CSI Plugin provides the creation and configuration of persistent storage external to Kubernetes. CSI replaces plugins developed earlier in the Kubernetes evolution. It replaces the hostPath method to expose WekaFS mounts as Kubernetes volumes.
Interoperability
CSI protocol: 1.0-1.2
Kubernetes: 1.18
OS (of Kubernetes worker nodes): ubuntu 16.04, 18.04
WekaFS: 3.8 and up
Note: For additional Kubernetes/OS versions support contact the Weka support team
Prerequisites
Privileged mode must be allowed on the Kubernetes cluster
The following Kubernetes feature gates must be enabled: DevicePlugins, CSINodeInfo, CSIDriverRegistry, ExpandCSIVolumes (if not changed, they should be enabled by default)
A Weka cluster is installed and accessible from the Kubernetes worker nodes
The Weka client is installed on the Kubernetes worker nodes
It is recommended to use a Weka client which is part of the cluster rather than a stateless client​
If the Kubernetes nodes are part of the Weka cluster (converged mode on the Weka servers), make sure the Weka processes come up before kubelet
Filesystems are pre-configured on the Weka system
Capabilities
Supported capabilities
Static and dynamic volumes provisioning
Mounting a volume as a WekaFS filesystem directory
All volume access modes are supported: ReadWriteMany, ReadWriteOnce, and ReadOnlyMany
Volume expansion
Unsupported capabilities
Snapshots
Deployment
The Weka CSI Plugin deployment can be performed with a Helm chart or directly as a DaemonSet (as described below).
Download
Download the Weka CSI Plugin from GitHub to a master node in the Kubernetes cluster.
1
$ git clone https://github.com/weka/csi-wekafs.git
Copied!
Installation
From the downloaded location in the Kubernetes master node, run the following command to deploy the Weka CSI Plugin as a DaemonsSet:
1
$ ./deploy/kubernetes-latest/deploy.sh
Copied!
On successful deployment, you will see the following output:
1
creating wekafsplugin namespace
2
namespace/csi-wekafsplugin created
3
deploying wekafs components
4
   ./deploy/kubernetes-latest/wekafs/csi-wekafs-plugin.yaml
5
        using           image: quay.io/k8scsi/csi-node-driver-registrar:v1.3.0
6
        using           image: quay.io/weka.io/csi-wekafs:v0.0.2-25-g7d18b61
7
        using           image: quay.io/k8scsi/livenessprobe:v1.1.0
8
        using           image: quay.io/k8scsi/csi-provisioner:v1.6.0
9
        using           image: quay.io/k8scsi/csi-attacher:v3.0.0-rc1
10
        using           image: quay.io/k8scsi/csi-resizer:v0.5.0
11
namespace/csi-wekafsplugin configured
12
csidriver.storage.k8s.io/wekafs.csi.k8s.io created
13
serviceaccount/csi-wekafsplugin created
14
clusterrole.rbac.authorization.k8s.io/csi-wekafsplugin-cluster-role created
15
clusterrolebinding.rbac.authorization.k8s.io/csi-wekafsplugin-cluster-role-binding created
16
role.rbac.authorization.k8s.io/csi-wekafsplugin-role created
17
rolebinding.rbac.authorization.k8s.io/csi-wekafsplugin-role-binding created
18
daemonset.apps/csi-wekafsplugin created
19
12:04:54 deployment completed successfully
20
12:04:54 2 plugin pods are running:
21
csi-wekafsplugin-dvdh2   6/6     Running   0          3h1m
22
csi-wekafsplugin-xh182   6/6     Running   0          3h1m
Copied!
The number of running pods should be the same as the number of Kubernetes worker nodes. This can be inspected by running:
1
$ kubectl get pods -n csi-wekafsplugin
2
NAME                     READY   STATUS    RESTARTS   AGE
3
csi-wekafsplugin-dvdh2   6/6     Running   0          3h2m
4
csi-wekafsplugin-xh182   6/6     Running   0          3h2m
Copied!
Provision Usage
The Weka CSI Plugin supports both dynamic (persistent volume claim) and static (persistent volume) volume provisioning.
It is first required to define a storage class to use the Weka CSI Plugin.
Storage Class Example
csi-wekafs/examples/dynamic/storageclass-wekafs-dir.yaml
1
apiVersion: storage.k8s.io/v1
2
kind: StorageClass
3
metadata:
4
  name: storageclass-wekafs-dir
5
provisioner: csi.weka.io
6
reclaimPolicy: Delete
7
volumeBindingMode: Immediate
8
allowVolumeExpansion: true
9
parameters:
10
  volumeType: dir/v1
11
  filesystemName: podsFilesystem
Copied!
Storage Class Parameters
Parameter
Description
Limitation
filesystemName
The name of the Weka filesystem to create directories in as Kubernetes volumes
The filesystem should exist in the Weka cluster
Apply the StorageClass and check it has been created successfully:
1
# apply the storageclass .yaml file
2
$ kubectl apply -f storageclass-wekafs-dir.yaml
3
storageclass.storage.k8s.io/storageclass-wekafs-dir created
4
​
5
# check the storageclass resource has been created
6
$ kubectl get sc
7
NAME                           PROVISIONER         RECLAIMPOLICY   VOLUMEBINDINGMODE   ALLOWVOLUMEEXPANSION   AGE
8
storageclass-wekafs-dir        csi.weka.io         Delete          Immediate           true                   75s
Copied!
It is possible to define multiple storage classes with different filesystems.
Dynamic Provisioning
Using a similar storage class to the above, it is possible to define a persistent volume claim (PVC) for the pods.
Persistent Volume Claim Example
csi-wekafs/examples/dynamic/pvc-wekafs-dir.yaml
1
apiVersion: v1
2
kind: PersistentVolumeClaim
3
metadata:
4
  name: pvc-wekafs-dir
5
spec:
6
  accessModes:
7
    - ReadWriteMany
8
  storageClassName: storageclass-wekafs-dir
9
  volumeMode: Filesystem
10
  resources:
11
    requests:
12
      storage: 1Gi
Copied!
Persistent Volume Claim Parameters
Parameter
Description
Limitation
spec.accessModes
The volume access mode
ReadWriteMany, ReadWriteOnce, or ReadOnlyMany
spec.storageClassName
The storage class to use to create the PVC
Must be an existing storage class
spec.resources.requests.storage
A desired capacity for the volume
The capacity quota is not enforced but is stored on the filesystem directory extended attributed for future use
Apply the PersistentVolumeClaim and check it has been created successfully:
1
# apply the pvc .yaml file
2
$ kubectl apply -f pvc-wekafs-dir.yaml
3
persistentvolumeclaim/pvc-wekafs-dir created
4
​
5
# check the pvc resource has been created
6
$ kubectl get pvc
7
NAME                  STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS                   AGE
8
pvc-wekafs-dir        Bound    pvc-d00ba0fe-04a0-4916-8fea-ddbbc8f43380   1Gi        RWX            storageclass-wekafs-dir        2m10s
Copied!
Note: The directory will be created inside the filesystem under csi-volumes directory, starting with the volume name.
Static Provisioning
The Kubernetes admin can prepare some persistent volumes in advance to be used by pods, they should be an existing directory, and can contain pre-populated data to be used by the PODs.
It can be a directory previously provisioned by the CSI or a pre-existing directory in WekaFS. To expose an existing directory in WekaFS via CSI, define a persistent volume, and link a persistent volume claim to this persistent volume.
Persistent Volume Example
csi-wekafs/examples/static/pv-wekafs-dir-static.yaml
1
apiVersion: v1
2
kind: PersistentVolume
3
metadata:
4
  name: pv-wekafs-dir-static
5
spec:
6
  storageClassName: storageclass-wekafs-dir
7
  accessModes:
8
    - ReadWriteMany
9
  persistentVolumeReclaimPolicy: Retain
10
  volumeMode: Filesystem
11
  capacity:
12
    storage: 1Gi
13
  csi:
14
    driver: csi.weka.io
15
    # volumeHandle must be formatted as following:
16
    # dir/v1/<FILE_SYSTEM_NAME>/<INNER_PATH_IN_FILESYSTEM>
17
    # The path must exist, otherwise publish request will fail
18
    volumeHandle: dir/v1/podsFilesystem/my-dir
Copied!
Persistent Volume Parameters
Parameter
Description
Limitation
spec.accessModes
The volume access mode
ReadWriteMany, ReadWriteOnce, or ReadOnlyMany
spec.storageClassName
The storage class to use to create the PV
Must be an existing storage class
spec.capacity.storage
A desired capacity for the volume
The capacity quota is not enforced but is stored on the filesystem directory extended attributed for future use
spec.csi.volumeHandle
A string specifying a previously created path
A string containing the volumeType (dir/v1) filesystem name, and the directory path. E.g. dir/v1/podsFilesystem/my-dir
Must be an existing filesystem and path
Apply the PersistentVolume and check it has been created successfully:
1
# apply the pv .yaml file
2
$ kubectl apply -f pv-wekafs-dir-static.yaml
3
persistentvolume/pv-wekafs-dir-static created
4
​
5
# check the pv resource has been created
6
$ kubectl get pv
7
NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS      CLAIM                         STORAGECLASS                   REASON   AGE
8
pv-wekafs-dir-static                       1Gi        RWX            Retain           Available                                 storageclass-wekafs-dir                 3m33s
Copied!
Now, bind a PVC to this specific PV, use the volumeName parameter under the PVC spec and provide it with the specific PV name.
Persistent Volume Claim for Static Provisioning Example
csi-wekafs/examples/static/pvc-wekafs-dir-static.yaml
1
apiVersion: v1
2
kind: PersistentVolumeClaim
3
metadata:
4
  name: pvc-wekafs-dir-static
5
spec:
6
  accessModes:
7
    - ReadWriteMany
8
  storageClassName: storageclass-wekafs-dir
9
  volumeName: pv-wekafs-dir-static
10
  volumeMode: Filesystem
11
  resources:
12
    requests:
13
      storage: 1Gi
Copied!
Persistent Volume Claim for Static Provisioning Example
Parameter
Description
Limitation
spec.accessModes
The volume access mode
ReadWriteMany, ReadWriteOnce, or ReadOnlyMany
spec.storageClassName
The storage class to use to create the PVC
Must be the same storage class as the PV requested to bind in spec.volumeName
spec.resources.requests.storage
A desired capacity for the volume
The capacity quota is not enforced but is stored on the filesystem directory extended attributed for future use
spec.volumeName
A name of a pre-configured persistent volume
Must be an existing PV name
Apply the PersistentVolumeClaim and check it has been created successfully:
1
# apply the pvc .yaml file
2
$ kubectl apply -f pvc-wekafs-dir-static.yaml
3
persistentvolumeclaim/pvc-wekafs-dir-static created
4
​
5
# check the pvc resource has been created
6
$ kubectl get pvc
7
NAME                    STATUS   VOLUME                CAPACITY   ACCESS MODES   STORAGECLASS                   AGE
8
pvc-wekafs-dir-static   Bound    pv-wekafs-dir-static  1Gi        RWX            storageclass-wekafs-dir        3m41s
Copied!
The PV will change the status to Bound and state the relevant claim it is bounded to:
1
# check the pv resource has been created
2
$ kubectl get pv
3
NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS      CLAIM                           STORAGECLASS                   REASON   AGE
4
pv-wekafs-dir-static                       1Gi        RWX            Retain           Bound       default/pvc-wekafs-dir-static   storageclass-wekafs-dir                 6m30s
Copied!
Launching an Application using Weka as the POD's Storage
Now that we have a storage class and a PVC in place, we can configure the Kubernetes pods to provision volumes via the Weka system.
We'll take an example application that echos the current timestamp every 10 seconds, and provide it with the previously created pvc-wekafs-dir PVC.
Note that multiple pods can share a volume produced by the same PVC as long as the accessModes parameter is set to ReadWriteMany.
csi-wekafs/examples/dynamic/csi-app-on-dir.yaml
1
kind: Pod
2
apiVersion: v1
3
metadata:
4
  name: my-csi-app
5
spec:
6
  containers:
7
    - name: my-frontend
8
      image: busybox
9
      volumeMounts:
10
      - mountPath: "/data"
11
        name: my-csi-volume
12
      command: ["/bin/sh"]
13
      args: ["-c", "while true; do echo `date` >> /data/temp.txt; sleep 10;done"]
14
  volumes:
15
    - name: my-csi-volume
16
      persistentVolumeClaim:
17
        claimName: pvc-wekafs-dir # defined in pvc-wekafs-dir.yaml
Copied!
Now we will apply that pod:
1
$ kubectl apply -f csi-app-on-dir.yaml
2
pod/my-csi-app created
Copied!
Kubernetes will allocate a persistent volume and attach it to the pod, it will use a directory within the WekaFS filesystem as defined in the storage class mentioned in the persistent volume claim. The pod will be in Running status, and the temp.txt file will get updated with occasional date information.
1
$ kubectl get pod my-csi-app
2
NAME                      READY   STATUS              RESTARTS   AGE
3
my-csi-app                1/1     Running             0          85s
4
​
5
# if we go to a wekafs mount of this filesystem we can see a directory has been created
6
$ ls -l /mnt/weka/podsFilesystem/csi-volumes
7
drwxr-x--- 1 root root 0 Jul 19 12:18 pvc-d00ba0fe-04a0-4916-8fea-ddbbc8f43380-a1659c8a7ded3c3c05d6facffd69cbf79b95604c
8
​
9
# inside that directory, the temp.txt file from the running pod can be found
10
 $ cat /mnt/weka/podsFilesystem/csi-volumes/pvc-d00ba0fe-04a0-4916-8fea-ddbbc8f43380-a1659c8a7ded3c3c05d6facffd69cbf79b95604c/temp.txt
11
Sun Jul 19 12:50:25 IDT 2020
12
Sun Jul 19 12:50:35 IDT 2020
13
Sun Jul 19 12:50:45 IDT 2020
Copied!
Troubleshooting
Useful Commands
Here are some useful basic commands to check the status and debug the service:
1
# get all resources
2
kubectl get all --all-namespaces
3
​
4
# get all pods
5
kubectl get pods --all-namespaces -o wide
6
​
7
# get all k8s nodes
8
kubectl get nodes
9
​
10
# get storage classes
11
$ kubectl get sc
12
​
13
# get persistent volume claims 
14
$ kubectl get pvc
15
​
16
# get persistent volumes
17
$ kubectl get pv
18
​
19
# kubectl describe pod/<pod-name> -n <namespace> 
20
kubectl describe pod/csi-wekafsplugin-dvdh2 -n csi-wekafsplugin
21
​
22
# get logs from a pod
23
kubectl logs <pod name> <container name>
24
​
25
# get logs from the weka csi plugin
26
# container (-c) can be one of: [node-driver-registrar wekafs liveness-probe csi-provisioner csi-attacher csi-resizer]
27
kubectl logs pods/csi-wekafsplugin-<ID> --namespace csi-wekafsplugin -c wekafs
Copied!
Known Issues
Mixed Hugepages Size Issue
Due to a Kubernetes v1.18 issue with allocating mixed hugepages sizes (https://github.com/kubernetes/kubernetes/pull/80831) is required that the Weka system will not try to allocate mixed sizes of hugepages on the Kubernetes nodes.
To workaround the Kubernetes issue (required only if the default memory for the client has been increased):
If the Weka client is installed on the K8s nodes via a manual stateless client mount, set the reserve_1g_hugepages mount option to false in the mount command.
If this is a Weka server or a Weka client, which is part of the Weka cluster, contact the Weka customer support team.