Set up protocols on K8s with WEKA Operator

Configure S3 and NFS protocols using the WEKA Kubernetes operator to provide object and file access to filesystems.

Overview of WEKA Operator protocols

The WEKA Kubernetes operator streamlines the deployment of protocol gateways by managing dedicated containers within the WekaCluster custom resource. By defining protocol settings in the cluster configuration, the operator automatically provisions and manages S3 and NFS frontend containers on the WEKA cluster fabric.

S3 protocol architecture

The S3 data path facilitates object access through a layered service structure:

• Client access: S3 clients communicate with the weka-s3 Service using the HTTP/S3 API.

• Load balancing: Traffic is directed to a Software Load Balancer (SLB Pod) running an SLB proxy.

• Protocol delivery: The SLB proxy routes requests to a pool of S3 Pods.

• Backend integration: S3 Pods communicate with the WEKA Backend using the internal WEKA protocol.

NFS protocol architecture

The NFS data path provides file access through a direct routing mechanism:

• Client access: NFS clients establish an NFS mount to the weka-nfs Service.

• Routing: The service routes traffic directly to the available NFS Pods.

• Backend integration: NFS Pods connect to the WEKA Backend by way of the WEKA protocol.

WEKA Operator S3 and NFS protocols setup

Before you begin

• Ensure the WEKA Operator is deployed and running in the Kubernetes cluster.

• Verify that the WekaCluster resource is initialized.

• Confirm that the target servers have sufficient CPU and memory resources to support additional protocol containers.

• Confirm the minimum versions of the WEKA Operator and WEKA image:

Protocol	Minimum WEKA Operator version	Minimum WEKA version
S3	1.7	4.4
NFS	1.10	5.1

Procedure

1. Open the WekaCluster YAML configuration file.

2. S3 protocol setup: Set the S3 parameters in the relevant sections.

   WekaCluster YAML configuration file example with S3 parameters

   apiVersion: weka.weka.io/v1
   kind: WekaCluster
   metadata:
     name: weka-cluster-dev
     namespace: default
   spec:
     gracefulDestroyDuration: "0"
     template: dynamic
     dynamicTemplate:
       s3Containers: 3
       s3Cores: 2
       s3FrontendHugepages: 3072
       envoyCores: 1
       computeContainers: 5 # WEKA cluster
       driveContainers: 5 # WEKA cluster
     additionalMemory:
       s3: 1000  
     image: quay.io/weka.io/weka-in-container-dev:5.1.0.547
     nodeSelector:
       weka.io/supports-backends: "true"
     driversDistService: "https://weka-drivers-dist.weka-operator-system.svc.cluster.local:60002"
     imagePullSecret: "quay-io-robot-secret"

3. NFS protocol setup: Set the NFS parameters in the relevant sections.

   WekaCluster YAML configuration file example with NFS parameters

   apiVersion: weka.weka.io/v1
   kind: WekaCluster
   metadata:
     name: weka-cluster-dev
     namespace: default
   spec:
     gracefulDestroyDuration: "0"
     nfs:
       interfaces: ["ens5"]
       ipRanges: ["10.0.1.1-10.0.1.10"]
     template: dynamic
     dynamicTemplate:
       computeContainers: 5 # WEKA cluster
       driveContainers: 5 # WEKA cluster
       nfsContainers: 2
       nfsCores: 2
       nfsFrontendHugepages: 3072
     additionalMemory:
       nfs: 1000
     image: quay.io/weka.io/weka-in-container-dev:5.1.0.547
     nodeSelector:
       weka.io/supports-backends: "true"
     driversDistService: "https://weka-drivers-dist.weka-operator-system.svc.cluster.local:60002"
     imagePullSecret: "quay-io-robot-secret"

4. Apply the updated configuration to the Kubernetes cluster:

   kubectl apply -f <cluster-config>.yaml

5. Identify the S3 port: kubectl get wekacluster <cluster name> -o jsonpath='{.status.ports.s3Port}'

S3 configuration reference

Use the following parameters in the WekaCluster spec to define S3 settings.

Parameter	Description
dynamicTemplate.s3Containers	Total number of S3 containers to be deployed. Data type: Integer Example: 2
dynamicTemplate.s3Cores	Number of CPU cores assigned to each S3 container process. Data type: Integer Example: 3
dynamicTemplate.s3FrontendHugepages	Hugepage memory for the S3 frontend in MiB. A minimum of 1600 MiB is required. Data Type: Integer Example: 3072
dynamicTemplate.envoyCores	Number of CPU cores assigned to the software load balancer container. Type: Integer Example: 3
additionalMemory.s3	Additional memory allocation in MiB for S3 containers, exceeding automatic calculations. Data type: Integer Example: 1000

NFS configuration reference

Use the following parameters to define NFS and networking settings.

Parameter	Description
nfs.interfaces	Restricted network interfaces for NFS traffic. Data type: List of strings Example: ["ens5"]
nfs.ipRanges	Floating IP addresses for client access, supporting CIDR or range formats. Data type: List of strings Example: ["10.0.1.1-10.0.1.10"]
dynamicTemplate.nfsContainers	Experimental count of NFS frontend containers to create. Data type: Integer Example: 2
dynamicTemplate.nfsCores	Number of CPU cores assigned to each NFS container process. Data type: Integer Example: 3
dynamicTemplate.nfsFrontendHugepages	Hugepage memory for the NFS frontend in MiB. A minimum of 1600 MiB is required. Data type: Integer Example: 3072
additionalMemory.nfs	Additional memory allocation in MiB for NFS containers, exceeding automatic calculations. Data type: Integer Example: 1000