Cluster scaling
Adjusting the size of a WEKA cluster ensures optimal performance and cost efficiency. Expand to meet growing workloads or shrink to reduce resources as demand decreases.
Expand a cluster
Cluster expansion enhances system resources and storage capacity while maintaining cluster stability. This procedure describes how to expand a WEKA cluster by increasing the number of compute and drive containers.
This procedure exemplifies an expansion of a cluster with 6 compute and 6 drive containers to a cluster with 7 compute and 7 drive containers. Each driveContainer has one driveCore.
Before you begin
Verify the following:
Ensure sufficient resources are available.
Ensure valid Quay.io credentials for WEKA container images.
Ensure access to the WEKA operator namespace.
Check the number of available Kubernetes nodes using
kubectl get nodes.Ensure all existing WEKA containers are in Running state.
Confirm your cluster is healthy with
weka status.
Procedure
Update the cluster configuration by increasing container value from previous value in your YAML file:
spec:
template: dynamic
dynamicTemplate:
computeContainers: 7 # Increase from previous value
driveContainers: 7 # Increase from previous value
computeCores: 1
driveCores: 1
numDrives: 1Apply the updated configuration:
Expected results
Total of 14 backend containers (7 compute + 7 drive).
All new containers show status as UP.
Weka status shows increased storage capacity.
Protection status remains Fully protected.
Troubleshooting
If containers remain in Pending state, verify available node capacity.
Check for sufficient resources across Kubernetes nodes.
Review WEKA operator logs for expansion-related issues.
Considerations
The number of containers cannot exceed available Kubernetes nodes.
Pending containers indicate resource constraints or node availability issues.
Each expansion requires sufficient system resources across the cluster.
If your cluster has resource constraints or insufficient nodes, container creation may remain in a pending state until additional nodes become available.
Scale up by adding drives
Add drives to an existing drive container by increasing its drive count and triggering a pod rotation. Use this procedure when you expand cluster capacity and need the operator to apply the new drive count to a running drive container.
The operator does not apply a changed drive count to a running pod on its own. To put the new count into effect, you increase numDrives and then change the podConfigVersion Helm value. Changing podConfigVersion triggers a controlled pod rotation, which recreates the drive containers with the updated configuration.
Before you begin
Confirm the target servers have the additional NVMe drives installed and available. make sure WekaPolicy is in place to sign drives.
Confirm the cluster is healthy and is not running an upgrade or another rotation.
Check the current helm value of
podConfigVersion. The default value is1. Record the current value so you set the next value correctly.Confirm you have adequate amount hugepages configured on each Linux machine. For details, see Configure HugePages for Kubernetes worker nodes.
Procedure
Edit WekaCluster configuration to change the drive count assigned to each drive container.
Trigger pod rotation by changing helm value
Expand an S3 cluster
Expanding an S3 cluster is necessary when additional storage or improved performance is required. Follow the steps below to expand the cluster while maintaining data availability and integrity.
Procedure
Update cluster YAML: Increase the number of S3 containers in the cluster YAML file and re-deploy the configuration. Example YAML update:
Apply the changes:
Verify new pods: Confirm that additional S3 and Envoy pods are created and running. Use the following command to list all pods:
Ensure two new S3 and Envoy pods appear in the output and are in the Running state.
Validate expansion: Verify the S3 cluster has expanded to include the updated number of containers. Check the cluster status and ensure no errors are present. Use these commands for validation:
Confirm the updated configuration reflects four S3 containers and all components are operational.
Shrink a cluster
A WEKA cluster shrink operation reduces compute and drive containers to optimize resources and system footprint. Shrinking may free resources, lower costs, align capacity with demand, or decommission infrastructure. Perform carefully to ensure data integrity and service availability.
Before you begin
Verify the following:
Cluster is in a healthy state before beginning.
The WEKA cluster is operational and with sufficient redundancy.
At least one hot spare configured for safe container removal.
Procedure
Modify the cluster configuration:
Apply the updated configuration:
Verify the desired state change:
Replace <cluster-name> with your specific value.
Remove specific containers:
Identify containers to remove
Delete the compute container:
Delete the drive container:
Verify cluster stability:
Check container status.
Monitor cluster health.
Verify data protection status.
Expected results
Reduced number of active containers and related pod.
Cluster status shows Running.
All remaining containers running properly.
Data protection maintained.
No service disruption.
Troubleshooting
If cluster shows degraded status, verify hot spare availability.
Check operator logs for potential issues.
Ensure proper container termination.
Verify resource redistribution.
Limitations
Manual container removal required.
Must maintain minimum required containers for protection level.
Hot spare needed for safe removal.
Cannot remove containers below protection requirement.
Related topics
Expand and shrink cluster resources
Increase client cores
When system demands increase, you may need to add more processing power by increasing the number of client cores. This procedure shows how to increase client cores from 1 to 2 cores to improve system performance while maintaining stability.
Prerequisites
Sufficient hugepage memory (1500MiB per core).
Procedure
Update the WekaClient object configuration in your client YAML file:
AWS DPDK on EKS is not supported for this configuration.
Apply the updated client configuration:
Verify the new client core is added:
Replace <cluster-name> with your specific value.
Delete all client container pods to trigger the reconfiguration:
Never force-delete WEKA pods. Force deletion removes the pod from the orchestration layer only, while the underlying container continues running untracked. See Deletion behavior.
Replace <client-name> and <ip-address> with your specific values.
Verify the client containers have restarted and rejoined the cluster:
Look for pods with your client name prefix to confirm they are in Running state.
Confirm the core increase in the WEKA cluster using the following commands :
Verification
After completing these steps, verify that:
All client pods are in Running state.
The CORES value shows 2 for client containers.
The clients have successfully rejoined the cluster.
The system status shows no errors using
weka status.
Troubleshooting
If clients fail to restart:
Ensure sufficient hugepage memory is available.
Check pod events for specific error messages.
Verify the client configuration in the YAML file is correct.
Increase backend cores
Increase the number of cores allocated to compute and drive containers to improve processing capacity for intensive workloads.
The following procedure exemplifies increase of the computeCores and driveCores from 1 to 2 cores.
Procedure
Modify the cluster YAML configuration to update core allocation:
Apply the updated configuration:
Verify the changes are applied to the cluster configuration:
Troubleshooting
If core values are not updated after applying changes:
Verify the YAML syntax is correct.
Ensure the cluster configuration was successfully applied.
Verify the HugePages configuration can accommodate the additional cores.
Check for any error messages in the cluster events:
Core allocation changes may require additional steps for full implementation.
Monitor cluster performance after making changes.
Consider testing in a non-production environment first.
Contact support if core values persist at previous settings after applying changes.
Last updated