Manually configure the WEKA cluster using the resources generator
Detailed workflow for manually configuring the WEKA cluster using the resources generator in a multi-container backend architecture.
Last updated
Detailed workflow for manually configuring the WEKA cluster using the resources generator in a multi-container backend architecture.
Last updated
Perform this workflow using the resources generator only if you are not using the automated WMS, WSA, or WEKA Configurator.
The resources generator generates three resource files on each server in the /tmp
directory: drives0.json
, compute0.json
, and frontend0.json
. Then, you create the containers using these generated files of the cluster servers.
Download the resources generator from the GitHub repository to your local server: .
Example:
Copy the resources generator from your local server to all servers in the cluster.
Example for a cluster with 8 servers:
To enable execution, change the mode of the resources generator on all servers in the cluster.
Example for a cluster with 8 servers:
Remove the default container
Generate the resource files
Create drive containers
Create a cluster
Configure the SSD drives
Create compute containers
Create frontend containers
Configure number of data and parity drives
Configure number of hot spares
Name the cluster
Command: weka local stop default && weka local rm -f default
Stop and remove the auto-created default container created on each server.
Command: resources_generator.py
To generate the resource files for the drive, compute, and frontend processes, run the following command on each backend server:
./resources_generator.py --net <net-devices> [options]
The resources generator allocates the number of cores, memory, and other resources according to the values specified in the parameters.
The best practice for resources allocation is as follows:
1 drive core per NVMe device (SSD).
2-3 compute cores per drive core.
1-2 frontend cores if deploying a protocol container. If there is a spare core, it is used for a frontend container.
Minimum of 1 core for the OS.
For a server with 24 cores and 6 SSDs, allocate 6 drive cores and 12 compute cores, and optionally you can use 2 cores of the remaining cores for the frontend container. The OS uses the remaining 4 cores.
Run the following command line:
./resources_generator.py --net eth1 eth2 --drive-dedicated-cores 6 --compute-dedicated-cores 12 --frontend-dedicated-cores 2
For a server with 14 cores and 6 SSDs, allocate 6 drive cores and 6 compute cores, and optionally you can use 1 core of the remaining cores for the frontend container. The OS uses the remaining 1 core.
Run the following command line:
./resources_generator.py --net eth1 eth2 --drive-dedicated-cores 6 --compute-dedicated-cores 6 --frontend-dedicated-cores 1
Parameters
compute-core-ids
Specify the CPUs to allocate for the compute processes. Format: space-separated numbers
compute-dedicated-cores
Specify the number of cores to dedicate for the compute processes.
The maximum available cores
compute-memory
Specify the total memory to allocate for the compute processes.
Format: value and unit without a space.
Examples: 1024B, 10GiB, 5TiB.
The maximum available memory
core-ids
Specify the CPUs to allocate for the WEKA processes. Format: space-separated numbers.
drive-core-ids
Specify the CPUs to allocate for the drive processes. Format: space-separated numbers.
drive-dedicated-cores
Specify the number of cores to dedicate for the drive processes.
1 core per each detected drive
drives
Specify the drives to use.
This option overrides automatic detection. Format: space-separated strings.
All unmounted NVME devices
frontend-core-ids
Specify the CPUs to allocate for the frontend processes. Format: space-separated numbers.
-
frontend-dedicated-cores
Specify the number of cores to dedicate for the frontend processes.
1
max-cores-per-container
Override the default maximum number of cores per container for IO processes (19). If provided, the new value must be lower.
19
minimal-memory
Set each container's hugepages memory to 1.4 GiB * number of IO processes on the container.
net
*
Specify the network devices to use. Format: space-separated strings.
no-rdma
Don't take RDMA support into account when computing memory requirements.
False
num-cores
Override the auto-deduction of the number of cores.
All available cores
path
Specify the path to write the resource files.
'.'
spare-cores
Specify the number of cores to leave for OS and non-WEKA processes.
1
spare-memory
Specify the memory to reserve for non-WEKA requirements.
Argument format: a value and unit without a space.
Examples: 10GiB, 1024B, 5TiB.
The maximum between 8 GiB and 2% of the total RAM
weka-hugepages-memory
Specify the memory to allocate for compute, frontend, and drive processes.
Argument format: a value and unit without a space.
Examples: 10GiB, 1024B, 5TiB.
The maximum available memory
Command: weka local setup container
For each server in the cluster, create the drive containers using the resources generator output file drives0.json
.
The drives JSON file includes all the required values for creating the drive containers. Only the path to the JSON resource file is required (before cluster creation, the optional parameter join-ips
is not relevant).
Parameters
resources-path
*
A valid path to the resource file.
Command: weka cluster create
To create a cluster of the allocated containers, use the following command:
Parameters
hostnames
*
Hostnames or IP addresses. If port 14000 is not the default for the drives, you can specify hostnames:port or ips:port. Minimum cluster size: 6 Format: space-separated strings
host-ips
IP addresses of the management interfaces. Use a list of ip+ip
addresses pairs of two cards for HA configuration. In case the cluster is connected to both IB and Ethernet, it is possible to set up to 4 management IPs for redundancy of both the IB and Ethernet networks using a list of ip+ip+ip+ip
addresses.
The same number of values as in hostnames
.
Format: comma-separated IP addresses.
IP of the first network device of the container
Command: weka cluster drive add
To configure the SSD drives on each server in the cluster, or add multiple drive paths, use the following command:
Parameters
container-id
*
The Identifier of the drive container to add the local SSD drives.
device-paths
*
List of block devices that identify local SSDs.
It must be a valid Unix network device name.
Format: Space-separated strings.
Example, /dev/nvme0n1 /dev/nvme1n1
Command: weka local setup container
For each server in the cluster, create the compute containers using the resources generator output file compute0.json
.
Parameters
resources-path
*
A valid path to the resource file.
join-ips
IP:port
pairs for the management processes to join the cluster. In the absence of a specified port, the command defaults to using the standard WEKA port 14000. Set the values, only if you want to customize the port.
To restrict the client’s operations to only the essential APIs for mounting and unmounting operations, connect to WEKA clusters through TCP base port + 3 (for example, 14003).
The IP:port
value must match the value used to create the container.
Format: comma-separated IP addresses.
Example: --join-ips 10.10.10.1,10.10.10.2,10.10.10.3:15000
Command: weka local setup container
For each server in the cluster, create the frontend containers using the resources generator output file frontend0.json
.
Command example for installing a stateful client with restricted privileges
Parameters
resources-path
*
A valid path to the resource file.
join-ips
IP:port pairs for the management processes to join the cluster. In the absence of a specified port, the command defaults to using the standard WEKA port 14000. Set the values, only if you want to customize the port.
Format: comma-separated IP addresses.
Example: --join-ips 10.10.10.1,10.10.10.2,10.10.10.3:15000
client
Set the container as a client.
auto-remove-timeout
Specify timeout (in seconds) for automatically removing inactive client containers. Only applicable when used with the --client
flag.
restricted
Set a client container with restricted privileges as a regular user regardless of the logged-in role.
Command: weka cluster update --data-drives=<count> --parity-drives=<count>
Example: weka cluster update --data-drives=4 --parity-drives=2
Command: weka cluster hot-spare <count>
Example: weka cluster hot-spare 1
Command: weka cluster update --cluster-name=<cluster name>