Manually prepare the system for WEKA configuration
If the system is not prepared using the WMS, perform this procedure to set the networking and other tasks before configuring the WEKA cluster.
Last updated
If the system is not prepared using the WMS, perform this procedure to set the networking and other tasks before configuring the WEKA cluster.
Last updated
Once the hardware and software prerequisites are met, prepare the backend servers and clients for the WEKA system configuration.
This preparation consists of the following steps:
(when required)
Related topics
Prerequisites and compatibility
To install Mellanox OFED, see .
To install Broadcom driver, see Broadcom adapter setup for WEKA system.
To install Intel driver, see .
Single Root I/O Virtualization (SR-IOV) enablement is mandatory in the following cases:
The servers are equipped with Intel NICs.
When working with client VMs where it is required to expose the virtual functions (VFs) of a physical NIC to the virtual NICs.
Related topic
The following example of the ifcfg
script is a reference for configuring the Ethernet interface.
For the best performance, MTU 9000 (jumbo frame) is recommended. For jumbo frame configuration, refer to your switch vendor documentation.
Bring the interface up using the following command:
InfiniBand network configuration normally includes Subnet Manager (SM), but the procedure involved is beyond the scope of this document. However, it is important to be aware of the specifics of your SM configuration, such as partitioning and MTU, because they can affect the configuration of the endpoint ports in Linux. For best performance, MTU of 4092 is recommended.
Refer to the following ifcfg
script when the IB network only has the default partition, i.e., "no pkey
":
Bring the interface up using the following command:
Verify that the “default partition” connection is up, with all the attributes set:
ignore-carrier
ignore-carrier
is a NetworkManager configuration option. When set, it keeps the network interface up even if the physical link is down. It’s useful when services need to bind to the interface address at boot.
Open the /etc/NetworkManager/NetworkManager.conf
file to edit it.
Under the [main]
section, add one of the following lines depending on the operating system:
For some versions of Rocky Linux, RHEL, and CentOS: ignore-carrier=*
For some other versions: ignore-carrier=<device-name1>,<device-name2>
.
Replace <device-name1>,<device-name2>
with the actual device names you want to apply this setting to.
Example for RockyLinux and RHEL 8.7:
Example for some other versions:
Restart the NetworkManager service for the changes to take effect.
Use a large-size ICMP ping to check the basic TCP/IP connectivity between the interfaces of the servers:
The-M do
flag prohibits packet fragmentation, which allows verification of correct MTU configuration between the two endpoints.
-s 8972
is the maximum ICMP packet size that can be transferred with MTU 9000, due to the overhead of ICMP and IP protocols.
The following steps provide guidance for configuring dual-network links with policy-based routing on Linux systems. Adjust IP addresses and interface names according to your environment.
/etc/sysctl.conf
Open the /etc/sysctl.conf
file using a text editor.
Add the following lines at the end of the file to set minimal configurations per InfiniBand (IB) or Ethernet (Eth) interface:
Save the file.
Apply the new settings by running:
Navigate to /etc/sysconfig/network-scripts/
.
Create the file /etc/sysconfig/network-scripts/route-mlnx0
with the following content:
Create the file /etc/sysconfig/network-scripts/route-mlnx1
with the following content:
Create the files /etc/sysconfig/network-scripts/rule-mlnx0
and /etc/sysconfig/network-scripts/rule-mlnx1
with the following content:
Open /etc/iproute2/rt_tables
and add the following lines:
Save the changes.
For Ethernet (ETH): To set up routing for Ethernet connections, use the following commands:
The route's first IP address in the provided commands represents the network's subnet to which the NIC is connected. The last address in the routing rules corresponds to the IP address of the NIC being configured, where eth1
is set to 10.10.10.1
.
For InfiniBand (IB): To configure routing for InfiniBand connections, use the following commands:
The route's first IP address in the above commands signifies the network's subnet associated with the respective NIC. The last address in the routing rules corresponds to the IP address of the NIC being configured, where ib0
is set to 10.10.10.1
.
Open the Netplan configuration file /etc/netplan/01-netcfg.yaml
and adjust it:
After adjusting the Netplan configuration file, run the following commands:
Create /etc/sysconfig/network/ifrule-eth2
with:
Create /etc/sysconfig/network/ifrule-eth4
with:
Create /etc/sysconfig/network/scripts/ifup-route.eth2
with:
Create /etc/sysconfig/network/scripts/ifup-route.eth4
with:
Add the weka lines to /etc/iproute2/rt_tables
:
Restart the interfaces or reboot the machine:
Related topic
The synchronization of time on computers and networks is considered good practice and is vitally important for the stability of the WEKA system. Proper timestamp alignment in packets and logs is very helpful for the efficient and quick resolution of issues.
Configure the clock synchronization software on the backends and clients according to the specific vendor instructions (see your OS documentation), before installing the WEKA software.
The WEKA system autonomously manages NUMA balancing, making optimal decisions. Therefore, turning off the Linux kernel’s NUMA balancing feature is a mandatory requirement to prevent extra latencies in operations. It’s crucial that the disabled NUMA balancing remains consistent and isn’t altered by a server reboot.
To persistently disable NUMA balancing, follow these steps:
Open the file located at: /etc/sysctl.conf
Append the following line: kernel.numa_balancing=disable
WEKA highly recommends that any servers used as backends have no swap configured. This is distribution-dependent but is often a case of commenting out any swap
entries in /etc/fstab
and rebooting.
The wekachecker
is a tool that validates the readiness of the servers in the cluster before installing the WEKA software.
The wekachecker
performs the following validations:
Dataplane IP, jumbo frames, and routing
ssh connection to all servers
Timesync
OS release
Sufficient capacity in /opt/weka
Available RAM
Internet connection availability
NTP
DNS configuration
Firewall rules
WEKA required packages
OFED required packages
Recommended packages
HT/AMT is disabled
The kernel is supported
CPU has a supported AES, and it is enabled
Numa balancing is enabled
RAM state
XFS FS type installed
Mellanox OFED is installed
IOMMU mode for SSD drives is disabled
rpcbind utility is enabled
SquashFS is enabled
noexec mount option on /tmp
Procedure
From the install directory, run ./wekachecker <hostnames/IPs>
Where:
The hostnames/IPs
is a space-separated list of all the cluster hostnames or IP addresses connected to the high-speed networking.
Example:
./wekachecker 10.1.1.11 10.1.1.12 10.1.1.4 10.1.1.5 10.1.1.6 10.1.1.7 10.1.1.8
Review the output. If failures or warnings are reported, investigate them and correct them as necessary. Repeat the validation until no important issues are reported.
The wekachecker
writes any failures or warnings to the file: test_results.txt
.
Once the report has no failures or warnings that must be fixed, you can install the WEKA software.
If you can use the WEKA Configurator, go to:
Configure the WEKA cluster using the WEKA Configurator
Otherwise, go to:
Manually configure the WEKA cluster using the resources generator
Download the wekachecker tarball from and extract it.