W E K A
4.3
4.3
  • WEKA v4.3 documentation
    • Documentation revision history
  • WEKA System Overview
    • WEKA Data Platform introduction
      • WEKA system functionality features
      • Converged WEKA system deployment
      • Optimize redundancy in WEKA deployments
    • SSD capacity management
    • Filesystems, object stores, and filesystem groups
    • WEKA networking
    • Data lifecycle management
    • WEKA client and mount modes
    • WEKA containers architecture overview
    • Glossary
  • Planning and Installation
    • Prerequisites and compatibility
    • WEKA cluster installation on bare metal servers
      • Plan the WEKA system hardware requirements
      • Obtain the WEKA installation packages
      • Install the WEKA cluster using the WMS with WSA
      • Install the WEKA cluster using the WSA
      • Manually install OS and WEKA on servers
      • Manually prepare the system for WEKA configuration
        • Broadcom adapter setup for WEKA system
        • Enable the SR-IOV
      • Configure the WEKA cluster using the WEKA Configurator
      • Manually configure the WEKA cluster using the resource generator
      • Perform post-configuration procedures
      • Add clients to an on-premises WEKA cluster
    • WEKA Cloud Deployment Manager Web (CDM Web) User Guide
    • WEKA Cloud Deployment Manager Local (CDM Local) User Guide
    • WEKA installation on AWS
      • WEKA installation on AWS using Terraform
        • Terraform-AWS-WEKA module description
        • Deployment on AWS using Terraform
        • Required services and supported regions
        • Supported EC2 instance types using Terraform
        • WEKA cluster auto-scaling in AWS
        • Detailed deployment tutorial: WEKA on AWS using Terraform
      • WEKA installation on AWS using the Cloud Formation
        • Self-service portal
        • CloudFormation template generator
        • Deployment types
        • AWS Outposts deployment
        • Supported EC2 instance types using Cloud Formation
        • Add clients to a WEKA cluster on AWS
        • Auto scaling group
        • Troubleshooting
      • Install SMB on AWS
    • WEKA installation on Azure
    • WEKA installation on GCP
      • WEKA project description
      • GCP-WEKA deployment Terraform package description
      • Deployment on GCP using Terraform
      • Required services and supported regions
      • Supported machine types and storage
      • Auto-scale instances in GCP
      • Add clients to a WEKA cluster on GCP
      • Troubleshooting
      • Detailed deployment tutorial: WEKA on GCP using Terraform
      • Google Kubernetes Engine and WEKA over POSIX deployment
  • Getting Started with WEKA
    • Manage the system using the WEKA GUI
    • Manage the system using the WEKA CLI
      • WEKA CLI hierarchy
      • CLI reference guide
    • Run first IOs with WEKA filesystem
    • Getting started with WEKA REST API
    • WEKA REST API and equivalent CLI commands
  • Performance
    • WEKA performance tests
      • Test environment details
  • WEKA Filesystems & Object Stores
    • Manage object stores
      • Manage object stores using the GUI
      • Manage object stores using the CLI
    • Manage filesystem groups
      • Manage filesystem groups using the GUI
      • Manage filesystem groups using the CLI
    • Manage filesystems
      • Manage filesystems using the GUI
      • Manage filesystems using the CLI
    • Attach or detach object store buckets
      • Attach or detach object store bucket using the GUI
      • Attach or detach object store buckets using the CLI
    • Advanced data lifecycle management
      • Advanced time-based policies for data storage location
      • Data management in tiered filesystems
      • Transition between tiered and SSD-only filesystems
      • Manual fetch and release of data
    • Mount filesystems
      • Mount filesystems from Single Client to Multiple Clusters (SCMC)
    • Snapshots
      • Manage snapshots using the GUI
      • Manage snapshots using the CLI
    • Snap-To-Object
      • Manage Snap-To-Object using the GUI
      • Manage Snap-To-Object using the CLI
    • Quota management
      • Manage quotas using the GUI
      • Manage quotas using the CLI
  • Additional Protocols
    • Additional protocol containers
    • Manage the NFS protocol
      • Supported NFS client mount parameters
      • Manage NFS networking using the GUI
      • Manage NFS networking using the CLI
    • Manage the S3 protocol
      • S3 cluster management
        • Manage the S3 service using the GUI
        • Manage the S3 service using the CLI
      • S3 buckets management
        • Manage S3 buckets using the GUI
        • Manage S3 buckets using the CLI
      • S3 users and authentication
        • Manage S3 users and authentication using the CLI
        • Manage S3 service accounts using the CLI
      • S3 rules information lifecycle management (ILM)
        • Manage S3 lifecycle rules using the GUI
        • Manage S3 lifecycle rules using the CLI
      • Audit S3 APIs
        • Configure audit webhook using the GUI
        • Configure audit webhook using the CLI
        • Example: How to use Splunk to audit S3
      • S3 supported APIs and limitations
      • S3 examples using boto3
      • Access S3 using AWS CLI
    • Manage the SMB protocol
      • Manage SMB using the GUI
      • Manage SMB using the CLI
  • Operation Guide
    • Alerts
      • Manage alerts using the GUI
      • Manage alerts using the CLI
      • List of alerts and corrective actions
    • Events
      • Manage events using the GUI
      • Manage events using the CLI
      • List of events
    • Statistics
      • Manage statistics using the GUI
      • Manage statistics using the CLI
      • List of statistics
    • Insights
    • System congestion
    • Security management
      • Obtain authentication tokens
      • KMS management
        • Manage KMS using the GUI
        • Manage KMS using the CLI
      • TLS certificate management
        • Manage the TLS certificate using the GUI
        • Manage the TLS certificate using the CLI
      • CA certificate management
        • Manage the CA certificate using the GUI
        • Manage the CA certificate using the CLI
      • Account lockout threshold policy management
        • Manage the account lockout threshold policy using GUI
        • Manage the account lockout threshold policy using CLI
      • Manage the login banner
        • Manage the login banner using the GUI
        • Manage the login banner using the CLI
      • Manage Cross-Origin Resource Sharing
    • User management
      • Manage users using the GUI
      • Manage users using the CLI
    • Organizations management
      • Manage organizations using the GUI
      • Manage organizations using the CLI
      • Mount authentication for organization filesystems
    • Expand and shrink cluster resources
      • Add a backend server
      • Expand specific resources of a container
      • Shrink a cluster
    • Background tasks
      • Set up a Data Services container for background tasks
      • Manage background tasks using the GUI
      • Manage background tasks using the CLI
    • Upgrade WEKA versions
  • Licensing
    • License overview
    • Classic license
  • Monitor the WEKA Cluster
    • Deploy monitoring tools using the WEKA Management Station (WMS)
    • WEKA Home - The WEKA support cloud
      • Local WEKA Home overview
      • Deploy Local WEKA Home v3.0 or higher
      • Deploy Local WEKA Home v2.x
      • Explore cluster insights and statistics
      • Manage alerts and integrations
      • Enforce security and compliance
      • Optimize support and data management
    • Set up the WEKAmon external monitoring
    • Set up the SnapTool external snapshots manager
  • Support
    • Get support for your WEKA system
    • Diagnostics management
      • Traces management
        • Manage traces using the GUI
        • Manage traces using the CLI
      • Protocols debug level management
        • Manage protocols debug level using the GUI
        • Manage protocols debug level using the CLI
      • Diagnostics data management
  • Best Practice Guides
    • WEKA and Slurm integration
      • Avoid conflicting CPU allocations
    • Storage expansion best practice
  • WEKApod
    • WEKApod Data Platform Appliance overview
    • WEKApod servers overview
    • Rack installation
    • WEKApod initial system setup and configuration
    • WEKApod support process
  • Appendices
    • WEKA CSI Plugin
      • Deployment
      • Storage class configurations
      • Tailor your storage class configuration with mount options
      • Dynamic and static provisioning
      • Launch an application using WEKA as the POD's storage
      • Add SELinux support
      • NFS transport failback
      • Upgrade legacy persistent volumes for capacity enforcement
      • Troubleshooting
    • Convert cluster to multi-container backend
    • Create a client image
    • Update WMS and WSA
    • BIOS tool
Powered by GitBook
On this page
  • Upgrade overview
  • What is a non-disruptive upgrade (NDU)
  • Upgrade workflow
  • 1. Verify system upgrade prerequisites
  • 2. Prepare the cluster for upgrade
  • 3. Prepare the backend servers for upgrade (optional)
  • 4. Upgrade the backend servers
  • 5. Upgrade the clients
  • 6. Check the status after the upgrade
  1. Operation Guide

Upgrade WEKA versions

Upgrade your WEKA system with the latest version.

PreviousManage background tasks using the CLINextLicense overview

Last updated 2 months ago

Upgrade overview

The WEKA upgrade process supports upgrades of a WEKA system using a non-disruptive upgrade (NDU).

When considering an upgrade, adhere to the following guidelines:

  • To upgrade to 4.3.Y, the minimum source version allowed is 4.2.1.

  • The upgrade must always be from an older version to a newer one. For instance, an upgrade from 4.2.X to 4.3.Y is permissible only if 4.3.Y was released after 4.2.X.

  • Stay within the same major version or upgrade from an older major version to its successor.

  • There might be exceptions, so always check the minimum and maximum versions supported for the upgrade. For information on the release dates and applicable versions for the upgrade, visit .

For example, upgrading from version 4.2.10 (released on April 12, 2024) to version 4.3.0 (released on April 23, 2024) is allowed. However, future LTS releases do not permit upgrading to version 4.3.0. In the following example, even though version 4.3.1 (released on May 20, 2024) is older than version 4.2.11 (released on May 10, 2024), the maximum version allowed for the upgrade is 4.2.10 or 4.3.0.

What is a non-disruptive upgrade (NDU)

In MCB architecture, each container serves a single type of process, drive, frontend, or compute function. Therefore, upgrading one container at a time (rolling upgrade) is possible while the remaining containers continue serving the clients.

Internal upgrade process

Once you run the upgrade command in ndu mode, the following occurs:

  1. Downloading the version and preparing all backend servers.

  2. Rolling upgrade of the drive containers.

  3. Rolling upgrade of the compute containers.

  4. Rolling upgrade of the frontend and protocol containers and the protocol gateways.

Related topics

WEKA containers architecture overview

Upgrade workflow

Adhere to the following considerations:

  • Protocol separation: Upgrading a WEKA cluster with a server used for more than one of the following protocols, NFS, SMB, or S3, is not permitted. If such a case arises, the upgrade process does not initiate and indicates the servers that require protocol separation. Contact the Customer Success Team to ensure only one additional protocol is installed on each server.

  • Legacy NFS protocol: If a legacy NFS protocol is implemented, contact the Customer Success Team. In this case, the upgrade is blocked.

  • NFS file-locking prerequisite before upgrade: Ensure the rpc.statd and rpc-statd-notifiy services are stopped on the WEKA servers. If not, run the following commands: systemctl disable rpc-statd.service systemctl disable srpc-statd-notify-service

  • S3 Cluster Creation: If you plan to create an S3 cluster, it’s crucial to ensure the upgrade process is complete and all containers are up before initiating the creation.

1. Verify system upgrade prerequisites

Ensure the environment meets the necessary prerequisites before proceeding with any system upgrade. The WEKA Upgrade Checker Tool automates these essential checks, comprehensively assessing the system’s readiness. Whether performing a single-version upgrade or a multi-hop upgrade, following this procedure is mandatory.

Summary of the WEKA Upgrade Checker Tool results:

  1. Passed checks (Green): The system meets all prerequisites for the upgrade.

  2. Warnings (Yellow): Address promptly to resolve potential issues.

  3. Failures (Red): Do not proceed; they may lead to data loss.

Sample list of the verification steps performed by the WEKA Upgrade Checker Tool

Multi-hop version upgrades:

After completing an upgrade, a background process initiates the conversion of metadata to a new format (in specific versions). This conversion may take several minutes before another upgrade can commence. To monitor the progress, use the weka status CLI command and check if a data upgrade task is RUNNING.

By diligently following this system readiness validation procedure, you can confidently proceed with system upgrades, minimizing risks and ensuring a smooth upgrade.

  • Prioritize running the WEKA Upgrade Checker 24 hours before any scheduled upgrades. This step is critical to identify and address any potential issues proactively.

  • Ensure passwordless SSH access is set up on all backend servers. This is crucial for the seamless execution of the Python script while running the WEKA Upgrade Checker.

Procedure

  1. Log in to one of the backend servers as a root user:

    • Access the server using the appropriate credentials.

  2. Obtain the WEKA Upgrade Checker: Choose one of the following methods:

    • Method A: Direct download

      • Clone the WEKA Upgrade Checker GIT repository with the command: git clone https://github.com/weka/tools.git

    • Method B: Update from existing tools repository

      • If you have previously downloaded the tools repository, navigate to the tools directory.

      • Run git pull to update the tools repository with the latest enhancements. (The WEKA tools, including the WEKA Upgrade Checker, continuously evolve.)

  3. Run the WEKA Upgrade Checker: Navigate to the weka_upgrade_checker directory. It includes a binary version and a Python script of the tool. A minimum of Python 3.8 is required if you run the Python script.

    • Run the Python script:

      python3.8 ./weka_upgrade_checker.py --target-version <version>

      Or

    • Run the Python precompiled script: ./weka_upgrade_checker --target-version <version>

    Replace <version> with your target version. For example 4.4.4.

    The tool scans the backend servers and verifies the upgrade prerequisites.

  4. Review the results:

    • Pay attention to the following indicators:

      • Green: Passed checks. Ensure the tool's version is the latest.

      • Yellow: Warnings that require attention and remedy.

      • Red: Failed checks. If any exist, do not proceed. Contact the Customer Success Team.

  5. Send the log file to the Customer Success Team:

    • The weka_upgrade_checker.log is located in the same directory where you ran the tool. Share the latest log file with the Customer Success Team for further analysis.

2. Prepare the cluster for upgrade

Download the new WEKA version to one of the backend servers using one of the following methods depending on the cluster deployment:

  • Method A: Using a distribution server

  • Method B: Direct download and install from get.weka.io

  • Method C: If the connectivity to get.weka.io is limited

For details, select the relevant tab.

Use this method if the cluster environment includes a distribution server from which the target WEKA version can be downloaded.

If the distribution server contains the target WEKA version, run the following commands from the cluster backend server:

weka version get <version>
weka version prepare <version>

Where: <version> is the target WEKA version, for example: 4.3.0.

Example:

weka version get <version> --from https://[GET.WEKA.IO-TOKEN]@get.weka.io
weka version prepare <version>
  1. Select the Install tab.

  2. From the backend server, run the curl command line as shown in the following example.

  1. Download the new version tar file to a location from which you copy it to a dedicated directory in the cluster backend server, and untar the file.

  2. From the dedicated directory in the cluster backend server, run the install.sh command.

3. Prepare the backend servers for upgrade (optional)

When working with many backend servers, preparing them separately from the upgrade process in advance is possible to minimize the total upgrade time. For a small number of backend servers, this step is not required.

The preparation phase prepares all the connected backend servers for the upgrade, which includes downloading the new version and getting it ready to be applied.

Once the new version is downloaded to one of the backend servers, run the following CLI command:

weka local run --container drives0 --in <new-version> upgrade --prepare-only

Where:

<new-version>: Specify the new version. For example,4.3.4.

4. Upgrade the backend servers

Once a new software version is installed on one of the backend servers, upgrade the entire cluster backend servers to the new version by running the following command on the backend server.

If you already ran the preparation step, the upgrade command skips the download and preparation operations.

weka local run --container drives0 --in <new-version> upgrade

Consider the following guidelines:

  • Before switching the cluster to the new software release, the upgrade command distributes the new release to all cluster servers. It makes the necessary preparations, such as compiling the new wekafs driver.

  • If a failure occurs during the preparation, such as a disconnection of a server or failure to build a driver, the upgrade process stops, and a summary message indicates the problematic server.

  • If cleanup issues occur during a specific upgrade phase, rerun it with the relevant option:

    --ndu-drives-phase
    --ndu-frontends-phase
    --ndu-computes-phase
  • If the container running the upgrade process uses a port other than the default (14000), include the option --mgmt-port <existing-port> to the command.

5. Upgrade the clients

Once all backends are upgraded, the clients remain with the existing version and continue working with the upgraded backends. The client's version can only be one major version behind the version of the backends. Therefore, clients must be upgraded before the next cluster software version upgrade. The minimum source version for client upgrades is 4.2.X.

Stateless client upgrade options

  • If a stateless client is mounted on a single cluster, it is automatically upgraded to the backend version after rebooting, or a complete umount and mount is performed.

  • Stateless clients can also be upgraded manually.

  • Use the --client-only flag in the weka version get command to ensure that only the essential components relevant to the stateless client operation are downloaded, excluding non-relevant packages.

  • To limit the display of versions unless the complete set of components is present, use the --full flag with the weka version command This provides you with finer control over version information visibility.

  • You can manually upgrade the clients locally (one by one) or remotely (in batches), usually during a maintenance window.

Persistent client upgrade options

  • Clients can be upgraded manually. This can be done either locally on each client individually or remotely in batches. This process typically occurs during a scheduled maintenance window.

  • An upgrade is performed on a gateway, which is a persistent client that runs a specific protocol. This gateway is associated with containers with the allow_protocols parameter set to true. The upgrade process involves interaction with backend servers.

Client upgrade procedures

To upgrade a stateless or persistent client locally, connect to the client and run the following command line:

  1. Run: weka version get <target-version> --from <backend name or IP>:<port>

  2. Upgrade the agent by running the following (replace the x with the latest minor version): weka version set --agent-only 4.3.x

  3. Upgrade the client containers. Do one the following following:

    • For clients connected to a single cluster, run weka local upgrade

    • For clients connected to a multiple clusters, upgrade all containers simultaneously by running weka local upgrade --all

An alert is raised if there is a mismatch between the clients' and the cluster versions.

Add the --from <backend name or IP> option to download the client package only from the backend, thus avoiding downloading from get.weka.io. The default port is 14000.

To upgrade stateless or persistent clients remotely in batches, add the following options to the weka local upgrade command:

  • --mode=clients-upgrade: This option activates the remote upgrade.

  • --client-rolling-batch-size: This option determines the number of clients to upgrade in each batch. For example, if there are 100 clients, you can set this option to 10 and the upgrade will run 10 batches of 10 clients each.

If you need upgrade specific clients, add the --clients-to-upgrade and the clients' ids to upgrade. For example, --clients-to-upgrade 33,34,34.

If you need to skip upgrade of specific clients, add the --drop-host and the clients' ids to skip. For example, --drop-host 22,23.

If an upgrade of a client part of a batch fails, it stops the following batch upgrade. The current running batch continues the upgrade.

Command syntax

weka local run -C <backend name> --in <target release> upgrade --mode=clients-upgrade --client-rolling-batch-size <number of clients in a batch> --clients-to-upgrade <comma separated clients' ids> --drop-host <comma separated clients' ids> --from backends

Example

The following command line upgrade two clients in two batches (each batch has one client):

weka local run -C drives0 --in 4.2.0.78 upgrade --mode=clients-upgrade --client-rolling-batch-size 1

Output example:

6. Check the status after the upgrade

Once the upgrade is complete, verify that the cluster is in the new version by running the weka status command.

Example: The following is returned when the system is upgraded to version 4.3.0:

# weka status Weka v4.3.0 ...

The source system must be set up in MCB architecture. If not, contact the to convert the cluster architecture to MCB. See Convert cluster to multi-container backend. This workflow is only intended for professional services.

Customers running WEKA clusters on AWS with auto-scaling groups must contact the before converting the cluster to MCB.

Some background tasks, such as snapshot uploads or downloads, must be postponed or aborted. See the in the upgrade workflow for details.

Confirm that all backend servers meet the requirements of the target version. Address any discrepancies promptly.

Consult the topic for comprehensive guidance.

If the distribution server does not contain the target WEKA version, add the option --from to the command, and specify the distribution site, along with the token.

Use this method if the cluster environment has connectivity to .

From the Public Releases on the , select the required release.

Use this method if the cluster environment does not have connectivity to , such as with private networks or dark sites.

If a stateless client is mounted on multiple clusters, the client container version is the same as the client-target-version in the cluster (see ).

prerequisites and compatibility
Background tasks
get.weka.io
get.weka.io
get.weka.io
get.weka.io
Mount filesystems from multiple clusters on a single client
prerequisites
Verify system upgrade prerequisites
Prepare the cluster for upgrade
Prepare the backend servers for upgrade (optional)
Upgrade the backend servers
Upgrade the clients
Check the status after the upgrade
get.weka.io
Demo: WEKA Upgrade Checker
Customer Success Team
Customer Success Team
Releases example on get.weka.io
NDU process at a glance
Example: Install tab
Example: Download tab
Upgrade one client per batch