Troubleshoot WEKA Operator deployments

Diagnose and resolve common issues encountered during WEKA Operator deployment and operation.

Pod stuck in pending state

Symptom: A WekaContainer pod remains in Pending state and does not progress.

Procedure

1. Describe the pod to identify the scheduling issue:

kubectl describe pod <pod-name> -n <namespace>

1. Review the Events section of the output and match the condition to a resolution:

Common pending conditions and resolutions

Condition	Cause	Resolution
Pod blocked on weka.io/drives	The operator cannot allocate the required drives for the WekaContainer. Either more drives were requested than are available on the node, or too many driveContainers are already running.	Verify that drives are signed and that the number of drives requested in the WekaCluster spec does not exceed those available on the target nodes.
Image pull failure	The imagePullSecret is missing, incorrect, or not present in the target namespace.	Verify that a valid robot secret for quay.io exists in every namespace where WEKA resources are deployed. Each deployment requires a unique secret.
Insufficient resources	The node does not have enough CPU, memory, or HugePages to schedule the pod.	Verify node resource availability and confirm HugePages are configured correctly. See [Configure HugePages for Kubernetes worker nodes].
No matching node	The pod's nodeSelector does not match any available node labels.	See WekaCluster or WekaClient not creating WekaContainers below.

WekaContainers staying in WaitForDrivers

Symptom: WekaContainers remain in WaitForDrivers state and do not progress to Starting.

Cause: The Drivers-Loader cannot obtain the kernel driver. This occurs when the driver distribution service is unreachable, the driver package is unavailable for the node's kernel version, or kernel headers are missing on the build server.

Procedure

1. Confirm the driver distribution service is reachable from the node:

kubectl exec -n <namespace> <weka-pod> -- curl -v <driversDistService-url>

1. Check the Drivers-Loader logs for connectivity or signature errors:

kubectl logs -n <namespace> -l app=weka-drivers-loader

1. Verify that driversDistService in the WekaCluster or WekaClient spec points to the correct endpoint:

kubectl get wekacluster <cluster-name> -o jsonpath='{.spec.driversDistService}'

1. For locally-built drivers, confirm that kernel headers matching the running kernel are installed on the build server:

uname -r
apt list --installed | grep linux-headers

1. If the driver package is missing for the specific kernel version, see Driver download failures.

WekaClient containers stuck in Terminating

Symptom: WekaClient pod remains in Terminating state for an extended period.

Cause: The container is in Draining state, waiting for active mounts to be released before shutdown. This is expected behavior during deletion. The pod does not complete termination until all mounts are unmounted by the applications using them.

Procedure

1. Check the container state:

kubectl get wekacontainer <container-name> -n <namespace> -o jsonpath='{.status.status}'

1. Identify active mounts on the node:

kubectl get wekacontainer <container-name> -n <namespace> -o jsonpath='{.status.stats.activeMounts}'

1. Coordinate with application owners to unmount active WEKA filesystems on the affected node before the container can terminate cleanly.

Do not force-delete a pod in Draining state. See Do not force-delete WEKA pods.

Do not force-delete WEKA pods

Force-deleting a WEKA pod does not remove the underlying WekaContainer resource. The operator immediately recreates the pod on the same node, and the root cause remains unresolved.

To move or remove a container, delete the WekaContainer resource:

kubectl delete wekacontainer <container-name> -n <namespace>

The operator then recreates the container, schedules a fresh pod, and applies the current nodeSelector rules. For client containers with active mounts, resolve the mounts before deleting the WekaContainer resource.

WekaCluster never reaching StartingIO

Symptom: The WekaCluster remains in WaitForDrives and does not progress to StartingIO.

Cause: The cluster has not received enough signed drives to meet the threshold defined in startIoConditions.minNumDrives.

Procedure

1. Check the current cluster status:

kubectl get wekacluster <cluster-name>

1. Verify that the sign-drives WekaPolicy has been applied:

kubectl get wekapolicy -n weka-operator-system

1. Confirm that the weka.io/drives extended resource is present on the target nodes:

kubectl describe node <node-name> | grep weka.io/drives

1. If weka.io/drives is missing, the drive discovery process has not completed. Verify that the sign-drives WekaPolicy nodeSelector matches the labels on the backend nodes:

kubectl get nodes --show-labels | grep weka.io/supports-backends

1. Re-apply the sign-drives policy if needed:

kubectl apply -f sign-drives.yaml

1. Confirm that the number of available drives meets or exceeds startIoConditions.minNumDrives in the WekaCluster spec.

WekaCluster or WekaClient not creating WekaContainers

Symptom: A WekaCluster or WekaClient is applied successfully but no WekaContainers are created.

Cause: The nodeSelector in the CR does not match any node labels in the cluster. The operator cannot schedule containers on nodes that do not satisfy the selector.

Procedure

1. Retrieve the nodeSelector from the WekaCluster or WekaClient:

kubectl get wekacluster <cluster-name> -o jsonpath='{.spec.nodeSelector}'
kubectl get wekaclient <client-name> -o jsonpath='{.spec.nodeSelector}'

1. Verify that at least one node carries the required labels:

kubectl get nodes --show-labels | grep weka.io/supports-backends
kubectl get nodes --show-labels | grep weka.io/supports-clients

1. If no nodes match, apply the correct label to the target nodes:

kubectl label nodes <node-name> weka.io/supports-backends=true
kubectl label nodes <node-name> weka.io/supports-clients=true

1. For CSI issues where the CSI plugin is not functioning, confirm that the nodeSelector on the CSI installation and the WekaClient match. A mismatch prevents the CSI node plugin from running on the same nodes as the WEKA client pods.

CSI not functioning

Symptom: Persistent Volume Claims are not being provisioned or mounted.

Cause: The nodeSelector on the CSI installation and the WekaClient do not match, preventing the CSI node plugin from co-locating with the WEKA client pods.

Procedure

1. Retrieve the nodeSelector from the WekaClient:

kubectl get wekaclient <client-name> -o jsonpath='{.spec.nodeSelector}'

1. Retrieve the nodeSelector from the CSI node plugin DaemonSet:

kubectl get daemonset -n <csi-namespace> -o jsonpath='{.spec.template.spec.nodeSelector}'

1. Ensure both selectors match. Update the CSI configuration or WekaClient spec to align them.

2. Verify that the CSI node pods are running on the same nodes as the WekaClient pods:

kubectl get pods -n <csi-namespace> -o wide
kubectl get pods -n <namespace> -l app=weka-client -o wide

Related topics

WekaCluster and WekaContainer lifecycle

WEKA Operator driver management

WEKA CRD API Reference