Mount a single WEKA client to multiple clusters simultaneously, optimizing data access and workload distribution.
Last updated
Mounting filesystems from a single WEKA client to multiple clusters provides several advantages:
Expanded cluster connectivity: A single client can connect to up to seven clusters simultaneously, increasing storage capacity and computational capabilities.
Unified data access: Provides a consolidated view of data across multiple clusters, simplifying access and management while improving data availability, flexibility, and resource efficiency.
Optimized workload distribution: Enables efficient workload distribution across clusters, supporting scalable applications and enhancing overall performance.
Seamless integration: WEKA’s SCMC feature ensures smooth and efficient integration for clients accessing multiple clusters.
The bandwidth division in SCMC is a universal consideration based on the specific NIC's bandwidth. It applies across various NIC types, including those using DPDK or specific models like the X500-T1.
During SCMC mounts, each active connection can use the bandwidth available on its associated NIC port. This is true during peak usage and idle cases. In scenarios where NICs are dual-ported, each connection operates independently, leveraging its dedicated port.
When working with low-bandwidth NICs such as the X500-T1, a 10Gb/s NIC, consider bandwidth calculations. In the context of SCMC, each container (representing connectivity to a different cluster) uses half of the available bandwidth (5Gb/s) for a shared port. Note that a dual-port NIC has a dedicated port for each container, optimizing bandwidth distribution. Keep these factors in mind for an optimal SCMC setup.
Ensure the following requirements are met:
All clusters that run in this configuration must be at least version 4.2.
All client containers in the WEKA client must run the same minor version, at least version 4.2. The client version must be the same as the cluster or, at most, one version earlier.
All client containers must be of the same type, persistent or stateless clients. Mixing different client types in a single WEKA client is not allowed.
Each client container must run on its port. The default ports are 14000, 14101, 14201, 14301, 14401, 14501, and 14601. Ensure these ports allow egress on the client and ingress on the cluster.
For DPDK, each client container must have 5 GB of free RAM, and it is recommended to have a dedicated CPU core to get optimal performance.
Mounting a filesystem without these requirements may fail or overload the WEKA client.
When a stateless client mounts a filesystem in a cluster, it creates a client container with the same version as provided by the cluster. Because there may be situations where some of the clusters run a different WEKA version than the others, such as during an upgrade, it is required to set the same client target version on all clusters. The client target version is retained regardless of the cluster upgrade.
The client target version must be consistent across all clusters. It can match the cluster version or be one major version earlier (regardless the minor), provided that version is available in the cluster for client download.
To upgrade the cluster to a version higher than the first major release above the client version, see Upgrade WEKA versions.
Connect to each cluster and run the following command to set the client target version.
Where: <version> is the designated client target version, which will be installed on the client container upon the mount command. Ensure this version is installed on the backend servers.
To display the existing client target version in the cluster, run the following command:
To reset the client target version to the cluster version, run the following command:
Use the same commands as with a single client.
To mount a stateless client using UDP mode, add -o net=udp -o core=<core-id> to the command line. For example:
For persistent client containers, the client-target-version parameter is not relevant. The version of the client container is determined when creating the container in the WEKA client using the weka local setup container command. Therefore, ensure that all client containers in the WEKA client have the same minor version as in the clusters.
To mount a persistent client container to a cluster, specify the container name for that mount.
When running WEKA CLI commands from a server hosting multiple client containers, each connected to a different WEKA cluster, it’s required to specify the client container port or the backend IP address/name of the cluster (linked to that client) in the command.
Consider a server with two client containers:
To run a WEKA CLI command on the second cluster (associated with client2), use either of the following methods:
By specifying the backend IP address or name linked to that client container (assuming the backend name is DataSphere2-1):
By specifying the client container port:
This approach ensures that your WEKA CLI command targets the correct WEKA cluster associated with the specified client container.
