Ctrlk
For the complete documentation index, see llms.txt. This page is also available as Markdown.

Cluster scaling

Adjusting the size of a WEKA cluster ensures optimal performance and cost efficiency. Expand to meet growing workloads or shrink to reduce resources as demand decreases.

Expand a cluster

Cluster expansion enhances system resources and storage capacity while maintaining cluster stability. This procedure describes how to expand a WEKA cluster by increasing the number of compute and drive containers.

This procedure exemplifies an expansion of a cluster with 6 compute and 6 drive containers to a cluster with 7 compute and 7 drive containers. Each driveContainer has one driveCore.

Before you begin

Verify the following:

  • Ensure sufficient resources are available.

  • Ensure valid Quay.io credentials for WEKA container images.

  • Ensure access to the WEKA operator namespace.

  • Check the number of available Kubernetes nodes using kubectl get nodes.

  • Ensure all existing WEKA containers are in Running state.

  • Confirm your cluster is healthy with weka status.

Procedure

  1. Update the cluster configuration by increasing container value from previous value in your YAML file:

cluster.yaml
spec:
  template: dynamic
  dynamicTemplate:
    computeContainers: 7  # Increase from previous value
    driveContainers: 7    # Increase from previous value
    computeCores: 1
    driveCores: 1
    numDrives: 1

  1. Apply the updated configuration:

Example

  1. Cluster scaling.

Expected results

  • Total of 14 backend containers (7 compute + 7 drive).

  • All new containers show status as UP.

  • Weka status shows increased storage capacity.

  • Protection status remains Fully protected.

Troubleshooting

  • If containers remain in Pending state, verify available node capacity.

  • Check for sufficient resources across Kubernetes nodes.

  • Review WEKA operator logs for expansion-related issues.

Considerations

  • The number of containers cannot exceed available Kubernetes nodes.

  • Pending containers indicate resource constraints or node availability issues.

  • Each expansion requires sufficient system resources across the cluster.

If your cluster has resource constraints or insufficient nodes, container creation may remain in a pending state until additional nodes become available.

Scale up by adding drives

Add drives to an existing drive container by increasing its drive count and triggering a pod rotation. Use this procedure when you expand cluster capacity and need the operator to apply the new drive count to a running drive container.

The operator does not apply a changed drive count to a running pod on its own. To put the new count into effect, you increase numDrives and then change the podConfigVersion Helm value. Changing podConfigVersion triggers a controlled pod rotation, which recreates the drive containers with the updated configuration.

Before you begin

  • Confirm the target servers have the additional NVMe drives installed and available. make sure WekaPolicy is in place to sign drives.

  • Confirm the cluster is healthy and is not running an upgrade or another rotation.

  • Check the current helm value of podConfigVersion. The default value is 1. Record the current value so you set the next value correctly.

  • Confirm you have adequate amount hugepages configured on each Linux machine. For details, see Configure HugePages for Kubernetes worker nodes.

Procedure

  1. Edit WekaCluster configuration to change the drive count assigned to each drive container.

  1. Trigger pod rotation by changing helm value

Expand an S3 cluster

Expanding an S3 cluster is necessary when additional storage or improved performance is required. Follow the steps below to expand the cluster while maintaining data availability and integrity.

Procedure

  1. Update cluster YAML: Increase the number of S3 containers in the cluster YAML file and re-deploy the configuration. Example YAML update:

    Apply the changes:

Example

  1. Verify new pods: Confirm that additional S3 and Envoy pods are created and running. Use the following command to list all pods:

Ensure two new S3 and Envoy pods appear in the output and are in the Running state.

Example

  1. Validate expansion: Verify the S3 cluster has expanded to include the updated number of containers. Check the cluster status and ensure no errors are present. Use these commands for validation:

Confirm the updated configuration reflects four S3 containers and all components are operational.

Example

Shrink a cluster

A WEKA cluster shrink operation reduces compute and drive containers to optimize resources and system footprint. Shrinking may free resources, lower costs, align capacity with demand, or decommission infrastructure. Perform carefully to ensure data integrity and service availability.

Before you begin

Verify the following:

  • Cluster is in a healthy state before beginning.

  • The WEKA cluster is operational and with sufficient redundancy.

  • At least one hot spare configured for safe container removal.

Procedure

  1. Modify the cluster configuration:

  1. Apply the updated configuration:

Example

  1. Verify the desired state change:

Replace <cluster-name> with your specific value.

Example

  1. Remove specific containers:

    • Identify containers to remove

    • Delete the compute container:

    • Delete the drive container:

  2. Verify cluster stability:

    • Check container status.

    • Monitor cluster health.

    • Verify data protection status.

Expected results

  • Reduced number of active containers and related pod.

  • Cluster status shows Running.

  • All remaining containers running properly.

  • Data protection maintained.

  • No service disruption.

Troubleshooting

  • If cluster shows degraded status, verify hot spare availability.

  • Check operator logs for potential issues.

  • Ensure proper container termination.

  • Verify resource redistribution.

Limitations

  • Manual container removal required.

  • Must maintain minimum required containers for protection level.

  • Hot spare needed for safe removal.

  • Cannot remove containers below protection requirement.

Related topics

Expand and shrink cluster resources

Increase client cores

When system demands increase, you may need to add more processing power by increasing the number of client cores. This procedure shows how to increase client cores from 1 to 2 cores to improve system performance while maintaining stability.

Prerequisites

Sufficient hugepage memory (1500MiB per core).

Procedure

  1. Update the WekaClient object configuration in your client YAML file:

AWS DPDK on EKS is not supported for this configuration.

  1. Apply the updated client configuration:

Example

  1. Verify the new client core is added:

Replace <cluster-name> with your specific value.

Example

  1. Delete all client container pods to trigger the reconfiguration:

Never force-delete WEKA pods. Force deletion removes the pod from the orchestration layer only, while the underlying container continues running untracked. See Deletion behavior.

Replace <client-name> and <ip-address> with your specific values.

Example for one node

  1. Verify the client containers have restarted and rejoined the cluster:

Look for pods with your client name prefix to confirm they are in Running state.

Example

  1. Confirm the core increase in the WEKA cluster using the following commands :

Example

Verification

After completing these steps, verify that:

  • All client pods are in Running state.

  • The CORES value shows 2 for client containers.

  • The clients have successfully rejoined the cluster.

  • The system status shows no errors using weka status.

Troubleshooting

If clients fail to restart:

  • Ensure sufficient hugepage memory is available.

  • Check pod events for specific error messages.

  • Verify the client configuration in the YAML file is correct.

Increase backend cores

Increase the number of cores allocated to compute and drive containers to improve processing capacity for intensive workloads.

The following procedure exemplifies increase of the computeCores and driveCores from 1 to 2 cores.

Procedure

  1. Modify the cluster YAML configuration to update core allocation:

  1. Apply the updated configuration:

Example

  1. Verify the changes are applied to the cluster configuration:

Example

Troubleshooting

If core values are not updated after applying changes:

  1. Verify the YAML syntax is correct.

  2. Ensure the cluster configuration was successfully applied.

  3. Verify the HugePages configuration can accommodate the additional cores.

  4. Check for any error messages in the cluster events:

  • Core allocation changes may require additional steps for full implementation.

  • Monitor cluster performance after making changes.

  • Consider testing in a non-production environment first.

  • Contact support if core values persist at previous settings after applying changes.

PreviousHardware maintenanceNextCluster maintenance

Last updated