W E K A
4.1
4.1
  • WEKA v4.1 documentation
  • WEKA System Overview
    • About the WEKA system
    • SSD capacity management
    • Filesystems, object stores, and filesystem groups
    • WEKA networking
    • Data lifecycle management
    • WEKA client and mount modes
    • WEKA containers architecture overview
    • Glossary
  • Getting Started with WEKA
    • Quick installation guide
    • Manage the system using the WEKA CLI
    • Manage the system using the WEKA GUI
    • Run first IOs with WEKA filesystem
    • Getting started with WEKA REST API
  • Planning and Installation
    • Prerequisites for installation
    • WEKA installation on bare metal
      • Plan the WEKA system Installation
      • Prepare the system for WEKA software installation
        • Enable the SR-IOV
      • Obtain the WEKA software installation package
      • WEKA cluster installation
        • WEKA legacy system installation process
      • Add clients
    • WEKA installation on AWS
      • Self-service portal
      • CloudFormation template generator
      • Deployment types
      • AWS outposts deployment
      • Supported EC2 instance types
      • Add clients
      • Auto scaling group
      • Troubleshooting
    • WEKA installation on Azure
    • WEKA installation on GCP
      • WEKA project description
      • Deployment on GCP using Terraform
      • GCP Terraform package description
      • Required services and supported regions
      • Supported machine types and storage
      • Auto-scale instances in GCP
      • Add clients
      • Troubleshooting
  • Performance
    • WEKA performance tests
      • Test environment details
  • WEKA Filesystems & Object Stores
    • Manage object stores
      • Manage object stores using the GUI
      • Manage object stores using the CLI
    • Manage filesystem groups
      • Manage filesystem groups using the GUI
      • Manage filesystem groups using the CLI
    • Manage filesystems
      • Manage filesystems using the GUI
      • Manage filesystems using the CLI
    • Attach or detach object store buckets
      • Attach or detach object store bucket using the GUI
      • Attach or detach object store buckets using the CLI
    • Advanced data lifecycle management
      • Advanced time-based policies for data storage location
      • Data management in tiered filesystems
      • Transition between tiered and SSD-only filesystems
      • Manual fetch and release of data
    • Mount filesystems
    • Snapshots
      • Manage snapshots using the GUI
      • Manage snapshots using the CLI
    • Snap-To-Object
      • Manage Snap-To-Object using the GUI
      • Manage Snap-To-Object using the CLI
    • Quota management
      • Manage quotas using the GUI
      • Manage quotas using the CLI
  • Additional Protocols
    • Manage the NFS protocol
      • Supported NFS client mount options
      • Manage NFS networking using the GUI
      • Manage NFS networking using the CLI
    • Manage the SMB protocol
      • Manage SMB using the GUI
      • Manage SMB using the CLI
    • Manage the S3 protocol
      • S3 cluster management
        • Manage the S3 service using the GUI
        • Manage the S3 service using the CLI
      • S3 buckets management
        • Manage S3 buckets using the GUI
        • Manage S3 buckets using the CLI
      • S3 users and authentication
        • Manage S3 users and authentication using the CLI
        • Manage S3 service accounts using the CLI
      • S3 rules information lifecycle management (ILM)
        • Manage S3 lifecycle rules using the GUI
        • Manage S3 lifecycle rules using the CLI
      • Audit S3 APIs
        • Configure audit webhook using the GUI
        • Configure audit webhook using the CLI
        • Example: How to use Splunk to audit S3
      • S3 supported APIs and limitations
      • S3 examples using boto3
  • Operation Guide
    • Alerts
      • Manage alerts using the GUI
      • Manage alerts using the CLI
      • List of alerts and corrective actions
    • Events
      • Manage events using the GUI
      • Manage events using the CLI
      • List of events
    • Statistics
      • Manage statistics using the GUI
      • Manage statistics using the CLI
      • List of statistics
    • System congestion
    • Security management
      • Obtain authentication tokens
      • KMS management
        • Manage KMS using the GUI
        • Manage KMS using the CLI
      • TLS certificate management
        • Manage the TLS certificate using the GUI
        • Manage the TLS certificate using the CLI
      • CA certificate management
        • Manage the CA certificate using the GUI
        • Manage the CA certificate using the CLI
      • Account lockout threshold policy management
        • Manage the account lockout threshold policy using GUI
        • Manage the account lockout threshold policy using CLI
      • Manage the login banner
        • Manage the login banner using the GUI
        • Manage the login banner using the CLI
    • User management
      • Manage users using the GUI
      • Manage users using the CLI
    • Organizations management
      • Manage organizations using the GUI
      • Manage organizations using the CLI
      • Mount authentication for organization filesystems
    • Expand and shrink cluster resources
      • Add a backend server in a multiple containers architecture
      • Add a backend server in a legacy architecture
      • Expand specific resources of a container
      • Shrink a cluster
    • Background tasks
    • Upgrade WEKA versions
  • Billing & Licensing
    • License overview
    • Classic license
    • Pay-As-You-Go license
  • Support
    • Prerequisites and compatibility
    • Get support for your WEKA system
    • Diagnostics management
      • Traces management
        • Manage traces using the GUI
        • Manage traces using the CLI
      • Protocols debug level management
        • Manage protocols debug level using the GUI
        • Manage protocols debug level using the CLI
      • Diagnostics data management
    • Weka Home - The WEKA support cloud
      • Local Weka Home overview
      • Local Weka Home deployment
      • Set the Local Weka Home to send alerts or events
      • Download the Usage Report or Analytics
  • Appendix
    • WEKA CSI Plugin
    • Set up the WEKAmon external monitoring
    • Set up the SnapTool external snapshots manager
  • REST API Reference Guide
Powered by GitBook
On this page
  • Install NIC drivers
  • Mellanox OFED installation
  • Enable SR-IOV
  • Configure the networking
  • Ethernet configuration
  • InfiniBand configuration
  • Verify the network configuration
  • Configure the HA networking
  • Configure the clock synchronization
  • Disable the NUMA balancing
  • Validate the system preparation
  1. Planning and Installation
  2. WEKA installation on bare metal

Prepare the system for WEKA software installation

This page describes the procedures required to system for the WEKA software installation.

PreviousPlan the WEKA system InstallationNextEnable the SR-IOV

Last updated 1 year ago

Once the hardware and software prerequisites are met, prepare the backend servers and clients for the installation of the WEKA system.

This preparation consists of the following steps:

  1. .

  2. (when required).

  3. .

  4. .

  5. .

  6. .

  7. .

Note: Some of the examples on this page contain version-specific information. The software is updated frequently, so the package versions available to you may differ from those presented here.

Related topics

Prerequisites and compatibility

Install NIC drivers

Note: The steps describing the installation of NIC drivers are provided as a courtesy. Refer to your NIC vendor documentation for the latest information and updates.

Mellanox OFED installation

This section describes an OFED installation procedure that has proven to be successful. However, Mellanox supports a number of other installation methods, any of which can be used to install OFED. For more information about other installation procedures, refer to the Mellanox documentation.

Meeting Mellanox OFED prerequisites

The Mellanox OFED installation has a number of dependencies. The following example shows the installation of OFED dependencies in RHEL/CentOS 7.x using yum's [base] and [update] repositories, which are supported and preconfigured in RHEL and CentOS.

yum install perl libnl lsof tcl libxml2-python tk

This example assumes that the server was provisioned using the "Minimal installation" option and that it has access to yum repositories, either locally or over the Internet. This method can trigger updates to existing packages already installed on the server.

Alternatively, it is possible to install OFED dependencies without triggering updates to already-installed packages, as shown in the following example:

yum --disablerepo=* --enablerepo=base install perl libnl lsof tcl libxml2-python tk

Once the dependencies have been satisfied, it is possible to perform the OFED installation procedure.

Mellanox OFED installation

The Mellanox OFED installation involves decompressing the distribution archive, which you obtain from the Mellanox website, and running the installation script. Refer to the following to begin the installation:

# tar xf MLNX_OFED_LINUX-4.5-1.0.1.0-rhel7.6-x86_64.tgz
# ls -lF
drwxr-xr-x   6 root root       4096 Nov 28  2018 MLNX_OFED_LINUX-4.5-1.0.1.0-rhel7.6-x86_64/
-rw-r--r--   1 root root  239624023 Dec  2  2018 MLNX_OFED_LINUX-4.5-1.0.1.0-rhel7.5-x86_64.tgz

# cd MLNX_OFED_LINUX-4.5-1.0.1.0-rhel7.6-x86_64/

# ./mlnxofedinstall
/tmp/MLNX_OFED_LINUX.414403.logs
General log file: /tmp/MLNX_OFED_LINUX.414403.logs/general.log
Verifying KMP rpms compatibility with target kernel...
This program will install the MLNX_OFED_LINUX package on your machine.
Note that all other Mellanox, OEM, OFED, RDMA or Distribution IB packages will be removed.
Those packages are removed due to conflicts with MLNX_OFED_LINUX, do not reinstall them.

Do you want to continue?[y/N]:y

On completion of the OFED installation, the NIC firmware may be updated to match the firmware requirements of the Mellanox OFED software. If an update was performed, reboot the server at the end of the installation for the new firmware to become effective. Otherwise, restart the driver by running the following:

# /etc/init.d/openibd restart

This concludes the Mellanox OFED installation procedure.

Enable SR-IOV

Single Root I/O Virtualization (SR-IOV) enablement is mandatory in the following cases:

  • The servers are equipped with Intel NICs.

  • When working with client VMs where it is required to expose the virtual functions (VFs) of a physical NIC to the virtual NICs.

Related topic

Enable the SR-IOV

Configure the networking

Ethernet configuration

The following example of ifcfg script is provided as a reference for configuring the Ethernet interface.

/etc/sysconfig/network-scripts/ifcfg-enp24s0
TYPE="Ethernet"
PROXY_METHOD="none"
BROWSER_ONLY="no"
BOOTPROTO="none"
DEFROUTE="no"
IPV4_FAILURE_FATAL="no"
IPV6INIT="no"
IPV6_AUTOCONF="no"
IPV6_DEFROUTE="no"
IPV6_FAILURE_FATAL="no"
IPV6_ADDR_GEN_MODE="stable-privacy"
NAME="enp24s0"
DEVICE="enp24s0"
ONBOOT="yes"
NM_CONTROLLED=no
IPADDR=192.168.1.1
NETMASK=255.255.0.0
MTU=9000

MTU 9000 (jumbo frame) is recommended for the best performance. Refer to your switch vendor documentation for jumbo frame configuration.

Bring the interface up using the following command:

# ifup enp24s0

InfiniBand configuration

InfiniBand network configuration normally includes Subnet Manager (SM), but the procedure involved is beyond the scope of this document. However, it is important to be aware of the specifics of your SM configuration, such as partitioning and MTU, because they can affect the configuration of the endpoint ports in Linux. For best performance, MTU of 4092 is recommended.

Refer to the following ifcfg script when the IB network only has the default partition, i.e., "no pkey":

/etc/sysconfig/network-scripts/ifcfg-ib1
TYPE=Infiniband
ONBOOT=yes
BOOTPROTO=static
STARTMODE=auto
USERCTL=no
NM_CONTROLLED=no
DEVICE=ib1
IPADDR=192.168.1.1
NETMASK=255.255.0.0
MTU=4092

Bring the interface up using the following command:

# ifup ib1

Verify that the “default partition” connection is up, with all the attributes set:

# ip a s ib1
4: ib1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 4092 qdisc mq state UP group default qlen 256
  link/infiniband 00:00:03:72:fe:80:00:00:00:00:00:00:24:8a:07:03:00:a8:09:48
brd 00:ff:ff:ff:ff:12:40:1b:ff:ff:00:00:00:00:00:00:ff:ff:ff:ff
    inet 10.0.20.84/24 brd 10.0.20.255 scope global noprefixroute ib0
       valid_lft forever preferred_lft forever

On an InfiniBand network with a non-default partition number, p-key must be configured on the interface if the InfiniBand ports on your network are members of an InfiniBand partition other than the default (0x7FFF). The p-key should associate the port as a full member of the partition (full members are those where the p-key number with the most-significant bit (MSB) of the 16-bits is set to 1).

Example: If the partition number is 0x2, the limited member p-key will equal the p-key itself, i.e.,0x2. The full member p-key will be calculated as the logical OR of 0x8000 and the p-key (0x2) and therefore will be equal to 0x8002.

Note: All InfiniBand ports communicating with the Weka cluster must be full members.

Two ifcfg scripts must be created for each pkey-ed IPoIB interface. To determine your own pkey-ed IPoIB interface configuration, refer to the following two examples where a pkey of 0x8002 is used:

/etc/sysconfig/network-scripts/ifcfg-ib1
TYPE=Infiniband
ONBOOT=yes
MTU=4092
BOOTPROTO=static
STARTMODE=auto
USERCTL=no
NM_CONTROLLED=no
DEVICE=ib1
/etc/sysconfig/network-scripts/ifcfg-ib1.8002
TYPE=Infiniband
BOOTPROTO=none
CONNECTED_MODE=yes
DEVICE=ib1.8002
IPV4_FAILURE_FATAL=yes
IPV6INIT=no
MTU=4092
NAME=ib1.8002
NM_CONTROLLED=no
ONBOOT=yes
PHYSDEV=ib1
PKEY_ID=2
PKEY=yes
BROADCAST=192.168.255.255
NETMASK=255.255.0.0
IPADDR=192.168.1.1

Bring the interface up using the following command:

# ifup ib1.8002

Verify the connection is up with all the non-default partition attributes set:

# ip a s ib1.8002
5: ib1.8002@ib0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 4092 qdisc mq state UP qlen 256
    link/infiniband 00:00:11:03:fe:80:00:00:00:00:00:00:24:8a:07:03:00:a8:09:48 brd 00:ff:ff:ff:ff:12:40:1b:80:02:00:00:00:00:00:00:ff:ff:ff:ff
    inet 192.168.1.1/16 brd 192.168.255.255 scope global noprefixroute ib1.8002
       valid_lft forever preferred_lft forever

Verify the network configuration

Use a large-size ICMP ping to check the basic TCP/IP connectivity between the interfaces of the servers:

# ping -M do -s 8972 -c 3 192.168.1.2
PING 192.168.1.2 (192.168.1.2) 8972(9000) bytes of data.
8980 bytes from 192.168.1.2: icmp_seq=1 ttl=64 time=0.063 ms
8980 bytes from 192.168.1.2: icmp_seq=2 ttl=64 time=0.087 ms
8980 bytes from 192.168.1.2: icmp_seq=3 ttl=64 time=0.075 ms

--- 192.168.2.0 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 1999ms
rtt min/avg/max/mdev = 0.063/0.075/0.087/0.009 ms

The-M do flag prohibits packet fragmentation, which allows verification of correct MTU configuration between the two endpoints.

-s 8972 is the maximum ICMP packet size that can be transferred with MTU 9000, due to the overhead of ICMP and IP protocols.

Configure the HA networking

Bonded interfaces are supported for ethernet can be added to WEKA after setting the bonded device in the server.

When there is a need to configure Dual Network (IB or ETH) without LACP, each NIC must have its own IP address and you will need to properly configure the routing of the interfaces involved.

Example using CentOS:

Add the following lines at the end of /etc/sysctl.conf:

net.ipv4.conf.ib0.arp_announce =2
net.ipv4.conf.ib1.arp_announce =2
net.ipv4.conf.ib0.arp_filter =1
net.ipv4.conf.ib1.arp_filter =1

This can be added per interface, as described above, or to all interfaces:

net.ipv4.conf.all.arp_filter = 1 
net.ipv4.conf.default.arp_filter = 1 
net.ipv4.conf.all.arp_announce = 2 
net.ipv4.conf.default.arp_announce = 2

Routing tables

Append the following to /etc/iproute2/rt_tables:

100 weka1
101 weka2

Assuming the interfaces are mlnx0 and mlnx1 and assuming that the network is 10.90.0.0/16 with IPs 10.90.0.1 and 10.90.1.1 and a default gw of 10.90.2.1, set the following routing rules:

/etc/sysconfig/network-scripts/route-mlnx0

10.90.0.0/16 dev mlnx0 src 10.90.0.1 table weka1
default via 10.90.2.1 dev mlnx0 table weka1

/etc/sysconfig/network-scripts/route-mlnx1

10.90.0.0/16 dev mlnx1 src 10.90.1.1 table weka2
default via 10.90.2.1 dev mlnx1 table weka2

/etc/sysconfig/network-scripts/rule-mlnx0

table weka1 from 10.90.0.1

/etc/sysconfig/network-scripts/rule-mlnx1

table weka2 from 10.90.1.1

Related topic

Configure the clock synchronization

The synchronization of time on computers and networks is considered good practice and is vitally important for the stability of the WEKA system. Proper timestamp alignment in packets and logs is very helpful for the efficient and quick resolution of issues.

Configure the clock synchronization software on the backends and clients according to the specific vendor instructions (see your OS documentation), before installing the WEKA software.

Disable the NUMA balancing

The WEKA system autonomously manages NUMA balancing, making optimal decisions. Therefore, turning off the Linux kernel’s NUMA balancing feature is a mandatory requirement to prevent extra latencies in operations. It’s crucial that the disabled NUMA balancing remains consistent and isn’t altered by a server reboot.

To persistently disable NUMA balancing, follow these steps:

  1. Open the file located at: /etc/sysctl.conf

  2. Append the following line: kernel.numa_balancing=disable

Validate the system preparation

The wekachecker is a tool that validates the readiness of the servers in the cluster before installing the WEKA software.

The wekachecker performs the following validations:

  • Dataplane IP, jumbo frames, and routing

  • ssh connection to all servers

  • Timesync

  • OS release

  • Sufficient capacity in /opt/weka

  • Available RAM

  • Internet connection availability

  • NTP

  • DNS configuration

  • Firewall rules

  • WEKA required packages

  • OFED required packages

  • Recommended packages

  • HT/AMT is disabled

  • The kernel is supported

  • CPU has a supported AES, and it is enabled

  • Numa balancing is enabled

  • RAM state

  • XFS FS type installed

  • Mellanox OFED is installed

  • IOMMU mode for SSD drives is disabled

  • rpcbind utility is enabled

  • SquashFS is enabled

  • noexec mount option on /tmp

Note: The wekacheckertool applies to all WEKA versions. From V4.0, the following validations are not relevant, although the tool displays them:

  • OS has SELinux disabled or in permissive mode.

  • Network Manager is disabled.

Procedure

  1. From the install directory, run ./wekachecker <hostnames/IPs> Where: The hostnames/IPs is a space-separated list of all the cluster hostnames or IP addresses connected to the high-speed networking. Example: ./wekachecker 10.1.1.11 10.1.1.12 10.1.1.4 10.1.1.5 10.1.1.6 10.1.1.7 10.1.1.8

  2. Review the output. If failures or warnings are reported, investigate them and correct them as necessary. Repeat the validation until no important issues are reported. The wekachecker writes any failures or warnings to the file: test_results.txt.

Once the report has no failures or warnings that must be fixed, you can install the WEKA software.

wekachecker report example

Dataplane IP Jumbo Frames/Routing test                       [PASS]
Check ssh to all hosts                                       [PASS]
Verify timesync                                              [PASS]
Check if OS has SELinux disabled or in permissive mode       [PASS]
Check OS Release...                                          [PASS]
Check /opt/weka for sufficient capacity...                   [WARN]
Check available RAM...                                       [PASS]
Check if internet connection available...                    [PASS]
Check for NTP...                                             [PASS]
Check DNS configuration...                                   [PASS]
Check Firewall rules...                                      [PASS]
Check for WEKA Required Packages...                          [PASS]
Check for OFED Required Packages...                          [PASS]
Check for Recommended Packages...                            [WARN]
Check if HT/AMT is disabled                                  [WARN]
Check if kernel is supported...                              [PASS]
Check if CPU has AES enabled and supported                   [PASS]
Check if Network Manager is disabled                         [WARN]
Checking if Numa balancing is enabled                        [WARN]
Checking RAM state for errors                                [PASS]
Check for XFS FS type installed                              [PASS]
Check if Mellanox OFED is installed                          [PASS]
Check for IOMMU disabled                                     [PASS]
Check for rpcbind enabled                                    [PASS]
Check for squashfs enabled                                   [PASS]
Check for /tmp noexec mount                                  [PASS]

RESULTS: 21 Tests Passed, 0 Failed, 5 Warnings

Refer to this to learn how to configure dual Ethernet network in RHEL

Download the wekachecker tarball from and extract it.

link
https://github.com/weka/tools/blob/master/install/wekachecker
Install NIC drivers
Enable SR-IOV
Configure the networking
Verify the network configuration
Configure the clock synchronization
Disable the Numa balancing
Validate the system preparation
High Availability (HA)