Expand and shrink cluster resources

Expand and shrink a cluster in a homogeneous WEKA system configuration.

A WEKA cluster is a collection of backend servers configured with containers, SSDs, cores, memory, and network resources. You can expand or shrink your cluster resources to meet the ongoing business changes.

• Expansion: To increase the performance and capacity of your cluster, you can expand your cluster by adding new servers to the cluster and configuring new resources anytime after installing and connecting them to the network on the same subnet as the cluster.

• Shrinking: If you want to save the cluster's costs and the performance degradation does not affect your business, you can shrink the cluster by removing SSDs and backend servers.

The expansion and shrinking procedures only apply to homogeneous WEKA clusters in which all cluster servers are similar and have the same number of cores, memory, SSD capacity per server, and servers per failure domain (if any).

Notes:

• For heterogeneous WEKA cluster configurations and estimation of the performance change, contact the Customer Success Team.

• For AWS deployments, use the CloudFormation for the initial deployment, not for expanding and shrinking cluster resources.

The expansion and shrinking procedures include:

• Add or delete backend servers and containers.

• Add or delete SSDs (drives).

• Modify the number of cores assigned to the WEKA cluster.

• Modify the memory size allocated to the WEKA cluster.

• Modify the network resources assigned to the WEKA cluster (not required frequently).

Expansion considerations

The expansion procedures are similar to the WEKA installation on bare metal procedures but require specific attention to the following considerations when planning the expansion:

• Containers architecture: The WEKA containers architecture must be retained. Two procedures for adding a backend server are provided:

  • Add a backend server in a multiple containers architecture.

  • Add a backend server in a single container architecture.

• Protection scheme: The WEKA cluster protection scheme is retained. You cannot modify it.

• Failure domains: Adding or removing failure domains are done automatically.

• Memory expansion: When expanding the memory resources, the new containers must have the same memory as the existing containers.

Note: To calculate the capacity of the WEKA cluster after the expansion, refer to the SSD net storage capacity calculation section.

Related topics

WEKA installation on bare metal

WEKA containers architecture overview

What happens after the expansion or shrinking?

Once the WEKA cluster expansion or shrinking is completed, the system starts a redistribution process. This involves redistributing all the existing data to be balanced between the original system SSDs and newly added SSDs.

The redistribution process time depends on the capacity and the networking CPU resources. It can take between minutes to hours.

The capacity increase is instant. Therefore, it is possible to define more filesystems immediately without waiting to complete the redistribution process.

When more containers or cores are expanded, the added CPU resources are operational in less than a minute. The write performance improves almost immediately, while the read performance only improves upon completing the data redistribution.