WEKA Operator best practices
Apply the following guidance to production WEKA Operator deployments to maximize reliability, performance, and operational clarity.
Preload images before deployment
Preloading WEKA container images on all target nodes before applying CRs eliminates download delays during cluster formation. Without preloading, nodes that complete their image pull early sit idle while others are still downloading, staggering the start of each deployment step.
Apply a DaemonSet to pull the image on every target node during environment preparation:
Example DaemonSet for image preloading
apiVersion: apps/v1
kind: DaemonSet
metadata:
name: weka-preload
namespace: default
spec:
selector:
matchLabels:
app: weka-preload
template:
metadata:
labels:
app: weka-preload
spec:
imagePullSecrets:
- name: quay-io-robot-secret
nodeSelector:
weka.io/supports-backends: "true"
tolerations:
- key: "key1"
operator: "Equal"
value: "value1"
effect: "NoSchedule"
- key: "key2"
operator: "Exists"
effect: "NoExecute"
containers:
- name: weka-preload
image: quay.io/weka.io/weka-in-container:<WEKA_IMAGE_VERSION_TAG>
command: ["sleep", "infinity"]
resources:
requests:
cpu: "100m"
memory: "128Mi"
limits:
cpu: "500m"
memory: "256Mi"Replace <WEKA_IMAGE_VERSION_TAG> with the target image version. Remove the DaemonSet after the deployment completes.
Configure control plane high availability
Use an odd number of etcd members to maintain quorum and tolerate failures. A cluster of N members tolerates up to (N−1)/2 failures.
For production backend deployments, use five or nine etcd members. Consider distributing control plane components across multiple failure domains or using an external etcd cluster. For more information, see the Kubernetes HA topology guidance.
Configure the Kubelet CPU Manager with static policy
Enable the static CPU Manager policy on all worker nodes to give WEKA processes dedicated CPU cores. Without this, the Kubernetes scheduler can place other workloads on the same cores, causing contention and reducing I/O throughput.
On Kubernetes v1.32 and later, also enable strict-cpu-reservation to prevent Burstable and Best Effort pods from scheduling onto reserved cores.
On hyperthreaded systems, reserve both logical CPUs of the physical core assigned to the OS. Reserving only one sibling leaves the physical core shared.
Identify HyperThreading sibling pairs on the node before configuring:
lscpu -e=cpu,core,socket,node
cat /sys/devices/system/cpu/cpu*/topology/thread_siblings_listApply the configuration to each worker node:
Adjust reservedSystemCPUs to match the sibling pairs reported by thread_siblings_list on each node.
Use consistent nodeSelector labels across all resources
Apply the same nodeSelector labels consistently across the WekaCluster, WekaClient, and CSI installation. A mismatch between these resources is one of the most common causes of WekaContainers not being created and CSI pods not co-locating with client pods.
WEKA recommends using the standard labels to simplify support and keep labeling consistent across the documentation, but you can use your own labels:
Backend nodes:
weka.io/supports-backends: "true"Client nodes:
weka.io/supports-clients: "true"
Verify labels are applied before deploying any CR:
Use pre-built drivers for standard deployments
Use the WEKA driver registry at https://drivers.weka.io for standard Linux distributions with supported kernel versions. This approach requires no build infrastructure and produces the fastest driver provisioning path.
Use a local Drivers-Builder only when the environment is air-gapped, uses a custom or patched kernel, or cannot reach the external registry. For locally-built drivers, ensure kernel headers matching the running kernel are installed on the build server before deployment:
Protect clusters against accidental deletion
The gracefulDestroyDuration field on the WekaCluster CR sets a protection window before the operator begins destroying the cluster after a deletion request. The default is 24 hours.
Keep the default in production environments. To cancel a deletion during the grace period, set overrides.cancelDeletion: true on the WekaCluster before the window expires:
Only reduce gracefulDestroyDuration to zero in non-production environments or when an immediate removal is intentional and confirmed.
Use rolling upgrade policy for WekaClient
Set upgradePolicy to rolling on all WekaClient CRs. This replaces client pods one at a time, preserving availability during upgrades. The all-at-once policy updates all pods simultaneously and causes a full client outage. The manual policy requires explicit pod deletion for each update and is prone to operator error.
Never force-delete WEKA pods
Never force-delete WEKA pods. Force deletion removes the pod from the orchestration layer only, while the underlying container continues running untracked.
Force-deleting a WEKA pod leaves the underlying container running on the node, holding all assigned resources and keeping WEKA processes active without the operator being aware of them. As a result, scheduling a replacement pod on the same node may fail, or resources may be over-allocated. The WekaContainer resource also remains, and the operator recreates the pod immediately, leaving the root cause unresolved.
To move or remove a container, delete the WekaContainer resource:
For client containers with active mounts, coordinate unmounting with application owners before deleting the WekaContainer resource.
Related topic
Monitor resources using custom display fields
WEKA custom resources expose additional status fields that are not visible in the default kubectl get output. Use the -o wide flag or k9s to access the full field set.
Assess WekaContainer status across all namespaces:
The STATUS and MODE columns provide a quick health overview. The DRIVES COUNT and WEKA CID columns confirm drive allocation and cluster membership per container.
Related topics
WekaCluster and WekaContainer lifecycle
Last updated