Upgrade WEKA versions

Upgrade overview

The WEKA upgrade process supports non-disruptive upgrades (NDUs) to ensure minimal impact on system operations. For a complete list of supported source and target versions, refer to the official compatibility information at get.weka.io.

Upgrade guidelines

Version requirements

• Upgrades must progress from older versions to newer versions.

• Version compatibility is based on release dates relative to LTS releases.

• Major version upgrades must be to the next consecutive version only.

• Confirm specific version compatibility at get.weka.io.

• Make sure the client's version is compatible with the backend upgrade version you intend to deploy. The client-target-version parameter must be consistently defined and identical across all clusters within a Single Client Multiple Clusters (SCMC). See Mount filesystems from Single Client to Multiple Clusters (SCMC).

Version compatibility rules

When upgrading from version 4.4.X to 5.0.Y:

• Version 5.0.Y must have been released after the 4.4.X LTS release.

• All intermediate versions must be supported versions.

Upgrade example

Supported upgrades

4.4.6.122 → 5.0.1.101    Maximum supported version
                         (released: May 15, 2025 → June 16, 2025)
4.4.6.114 → 5.0.1.101    Supported intermediate version
4.4.6     → 5.0.1.101    Minimum supported version

Unsupported upgrades

4.4.8.53  → 5.0.1.101     Version not in supported range
4.4.7.89  → 5.0.1.101     Version not in supported range (released after 5.0.1 code freeze)
4.4.4     → 5.0.1.101     Version not in supported range
4.4.3     → 5.0.1.101     Version not in supported range
...

Always review release notes for version-specific upgrade requirements.

What is a non-disruptive upgrade (NDU)

In MCB architecture, each container serves a single type of process, drive, frontend, or compute function. Therefore, upgrading one container at a time (rolling upgrade) is possible while the remaining containers continue serving the clients.

Internal upgrade process

Once you run the upgrade command in ndu mode, the following occurs:

1. Downloading the version and preparing all backend servers.

2. Rolling upgrade of the drive containers.

3. Rolling upgrade of the compute containers.

4. Rolling upgrade of the frontend and protocol containers and the protocol gateways.

Related topics

WEKA containers architecture overview

Upgrade workflow

1. Verify system upgrade prerequisites

2. Prepare the cluster for upgrade

3. Prepare the backend servers for upgrade (optional)

4. Upgrade the backend servers

5. Enable LLQ and WC in AWS

6. Upgrade the clients

7. Check the status after the upgrade

1. Verify system upgrade prerequisites

Ensure the environment meets the necessary prerequisites before proceeding with any system upgrade. The WEKA Upgrade Checker Tool automates these essential checks, comprehensively assessing the system’s readiness. Whether performing a single-version upgrade or a multi-hop upgrade, following this procedure is mandatory.

Summary of the WEKA Upgrade Checker Tool results:

1. Passed checks (Green): The system meets all prerequisites for the upgrade.

2. Warnings (Yellow): Address promptly to resolve potential issues.

3. Failures (Red): Do not proceed; they may lead to data loss.

By diligently following this system readiness validation procedure, you can confidently proceed with system upgrades, minimizing risks and ensuring a smooth upgrade.

Procedure

1. Log in to one of the backend servers as a root user:

   • Access the server using the appropriate credentials.

2. Obtain the WEKA Upgrade Checker: Choose one of the following methods:

   • Method A: Direct download

     • Clone the WEKA Upgrade Checker GIT repository with the command: git clone https://github.com/weka/tools.git

   • Method B: Update from existing tools repository

     • If you have previously downloaded the tools repository, navigate to the tools directory.

     • Run git pull to update the tools repository with the latest enhancements. (The WEKA tools, including the WEKA Upgrade Checker, continuously evolve.)

3. Run the WEKA Upgrade Checker: Navigate to the weka_upgrade_checker directory. It includes a binary version and a Python script of the tool. A minimum of Python 3.8 is required if you run the Python script.

   • Run the Python script:

     python3.8 ./weka_upgrade_checker.py --target-version <version>

   Or

   • Run the Python precompiled script: ./weka_upgrade_checker --target-version <version>

   Replace <version> with your target version. For example 4.4.4. The tool scans the backend servers and verifies the upgrade prerequisites.

4. Review the results:

   • Pay attention to the following indicators:

     • Green: Passed checks. Ensure the tool's version is the latest.

     • Yellow: Warnings that require attention and remedy.

     • Red: Failed checks. If any exist, do not proceed. Contact the Customer Success Team.

5. Send the log file to the Customer Success Team:

   • The weka_upgrade_checker.log is located in the same directory where you ran the tool. Share the latest log file with the Customer Success Team for further analysis.

2. Prepare the cluster for upgrade

Download the new WEKA version to one of the backend servers using one of the following methods depending on the cluster deployment:

• Method A: Using a distribution server

• Method B: Direct download and install from get.weka.io

• Method C: If the connectivity to get.weka.io is limited

For details, select the relevant tab.

Method A

Method B

Method C

Use this method if the cluster environment includes a distribution server from which the target WEKA version can be downloaded.

If the distribution server contains the target WEKA version, run the following commands from the cluster backend server:

Copy

weka version get <version>
weka version prepare <version>

Where: <version> is the target WEKA version, for example: 4.4.0.

If the distribution server does not contain the target WEKA version, add the option --from to the command, and specify the get.weka.io distribution site, along with the token.

Example:

Copy

weka version get <version> --from https://[GET.WEKA.IO-TOKEN]@get.weka.io
weka version prepare <version>

Use this method if the cluster environment has connectivity to get.weka.io.

1. From the Public Releases on the get.weka.io, select the required release.

2. Select the Install tab.

3. From the backend server, run the curl command line as shown in the following example.

Use this method if the cluster environment does not have connectivity to get.weka.io, such as with private networks or dark sites.

1. Download the new version tar file to a location from which you copy it to a dedicated directory in the cluster backend server, and untar the file.

2. From the dedicated directory in the cluster backend server, run the install.sh command.

3. Prepare the backend servers for upgrade (optional)

When working with many backend servers, preparing them separately from the upgrade process in advance is possible to minimize the total upgrade time. For a small number of backend servers, this step is not required.

The preparation phase prepares all the connected backend servers for the upgrade, which includes downloading the new version and getting it ready to be applied.

Once the new version is downloaded to one of the backend servers, run the following CLI command:

weka local run --container drives0 --in <new-version> upgrade --prepare-only

Where:

<new-version>: Specify the new version. For example,4.4.1.

4. Upgrade the backend servers

Once a new software version is installed on one of the backend servers, upgrade the entire cluster backend servers to the new version by running the following command on the backend server.

If you already ran the preparation step, the upgrade command skips the download and preparation operations.

weka local run --container drives0 --in <new-version> upgrade

Consider the following guidelines:

• Before switching the cluster to the new software release, the upgrade command distributes the new release to all cluster servers. It makes the necessary preparations, such as compiling the new wekafs driver.

• If a failure occurs during the preparation, such as a disconnection of a server or failure to build a driver, the upgrade process stops, and a summary message indicates the problematic server.

• If cleanup issues occur during a specific upgrade phase, rerun it with the relevant option:

  Copy

  --ndu-drives-phase
  --ndu-frontends-phase
  --ndu-computes-phase

• If the container running the upgrade process uses a port other than the default (14000), include the option --mgmt-port <existing-port> to the command.

5. Verify LLQ and WC are enabled in AWS

Enabling the Low Latency Queue (LLQ) improves data processing efficiency in AWS by reducing I/O operation delays. LLQ is enabled by default after an upgrade, but if Write Combining (WC) is not activated in the igb_uio driver, the LLQ driver option does not function. After upgrading the backends, verify that WC is enabled.

Procedure

1. Check for upgrade events:

   • Review the upgrade events on the backends.

   • If NetDevDriverReloadFailed appears, restart the WEKA service by running the following commands on each backend server:

     Copy

     weka local stop
     weka local start

2. Verify WC activation:

   • Check if WC is activated by running:

     Copy

     cat /sys/module/igb_uio/parameters/wc_activate

   • If the output is #1, WC is activated, which enables the LLQ driver option.

6. Upgrade the clients

After all backend components have been upgraded, clients continue operating with their existing version and can interact with the upgraded backends. Typically, a client version is supported only if it is no more than one major version behind the backend version. Therefore, clients must be upgraded before the subsequent cluster software version upgrade to maintain compatibility.

An exception to this rule is that clients running version 4.2.1 or later are compatible with clusters running version 4.4.6.

Stateless client upgrade options

• If a stateless client is mounted on a single cluster, it is automatically upgraded to the backend version after rebooting, or a complete umount and mount is performed.

• If a stateless client is mounted on multiple clusters, the client container version is the same as the client-target-version in the cluster (see Mount filesystems from multiple clusters on a single client).

• Stateless clients can also be upgraded manually.

• Use the --client-only flag in the weka version get command to ensure that only the essential components relevant to the stateless client operation are downloaded, excluding non-relevant packages.

• To limit the display of versions unless the complete set of components is present, use the --full flag with the weka version command This provides you with finer control over version information visibility.

• You can manually upgrade the clients locally (one by one) or remotely (in batches), usually during a maintenance window.

Persistent client upgrade options

• Clients can be upgraded manually. This can be done either locally on each client individually or remotely in batches. This process typically occurs during a scheduled maintenance window.

• An upgrade is performed on a gateway, which is a persistent client that runs a specific protocol. This gateway is associated with containers with the allow_protocols parameter set to true. The upgrade process involves interaction with backend servers.

Client upgrade procedures

Upgrade a client locally

Upgrade remote clients in batches

To upgrade a stateless or persistent client locally, connect to the client and run the following command line:

1. Run: weka version get <target-version> --from <backend name or IP>:<port>

2. Upgrade the agent by running the following (replace the x with the latest minor version): weka version set --agent-only 4.4.x

3. Upgrade the client containers. Do one the following following:

   • For clients connected to a single cluster, run weka local upgrade

   • For clients connected to a multiple clusters, upgrade all containers simultaneously by running weka local upgrade --all

An alert is raised if there is a mismatch between the clients' and the cluster versions.

Add the --from <backend name or IP> option to download the client package only from the backend, thus avoiding downloading from get.weka.io. The default port is 14000.

To upgrade stateless or persistent clients remotely in batches, add the following options to the weka local upgrade command:

• --mode=clients-upgrade: This option activates the remote upgrade.

• --client-rolling-batch-size: This option determines the number of clients to upgrade in each batch. For example, if there are 100 clients, you can set this option to 10 and the upgrade will run 10 batches of 10 clients each.

If you need upgrade specific clients, add the --clients-to-upgrade and the clients' ids to upgrade. For example, --clients-to-upgrade 33,34,34.

If you need to skip upgrade of specific clients, add the --drop-host and the clients' ids to skip. For example, --drop-host 22,23.

If an upgrade of a client part of a batch fails, it stops the following batch upgrade. The current running batch continues the upgrade.

Command syntax

weka local run -C <backend name> --in <target release> upgrade --mode=clients-upgrade --client-rolling-batch-size <number of clients in a batch> --clients-to-upgrade <comma separated clients' ids> --drop-host <comma separated clients' ids> --from backends

Example

The following command line upgrade two clients in two batches (each batch has one client):

weka local run -C drives0 --in 4.3.0.78 upgrade --mode=clients-upgrade --client-rolling-batch-size 1

Output example:

7. Check the status after the upgrade

Once the upgrade is complete, verify that the cluster is in the new version by running the weka status command.

Example: The following is returned when the system is upgraded to version 5.0.1:

# weka status
Weka v5.0.1
...