The WEKA system offers a converged deployment configuration as an alternative to the standard setup. In this configuration, hundreds of application servers running user applications are equipped with WEKA clients, allowing them to access the WEKA cluster.

Unlike the standard deployment that dedicates specific servers to WEKA backends, the converged setup involves installing a WEKA client on each application server. Additionally, one or more SSDs and backend processes (WekaFS) are integrated into the existing application servers.

The WEKA backend processes function collectively as a single, distributed, and scalable filesystem, leveraging the local SSDs. This filesystem is accessible to the application servers, much like in the standard WEKA system deployment. The critical distinction is that, in this configuration, WEKA backends share the same physical infrastructure as the application servers.

This blend of storage and computing capabilities enhances overall performance and resource usage. However, unlike the standard deployment, where an application server failure does not impact other backends, the converged setup is affected if an application server is rebooted or experiences a failure. The N+2 (or N+4) scheme still protects the cluster and can tolerate two concurrent failures. As a result, converged WEKA deployments require more careful integration and detailed coordination between computational and storage management practices.

In all other respects, this configuration mirrors the standard WEKA system, offering the same functionality features for protection, redundancy, failed component replacement, failure domains, prioritized data rebuilds, and seamless distribution, scale, and performance. Some servers may house a WEKA backend process and a local SSD, while others may have WEKA clients only. This allows for a cluster of application servers with a mix of WEKA software and WEKA clients, delivering a flexible solution.

Welcome to the WEKA Documentation Portal, your guide to the latest WEKA version. Whether you're a newcomer or a seasoned user, explore topics from system fundamentals to advanced optimization strategies. Choose your WEKA version from the top menu for version-specific documentation.

Get answers from WEKA documentation with Sevii AI

Sevii AI quickly delivers answers from WEKA documentation. Type your question and click . For the best results, ask clear, context-rich questions.

About WEKA documentation

This portal encompasses all documentation essential for comprehending and operating the WEKA system. It covers a range of topics:

WEKA system overview: Delve into the fundamental components, principles, and entities constituting the WEKA system.

Planning and installation: Discover prerequisites, compatibility details, and installation procedures for WEKA clusters on bare metal, AWS, GCP, and Azure environments.

Getting started with WEKA: Initiate your WEKA journey by learning the basics of managing a WEKA filesystem through the GUI and CLI, executing initial IOs, and exploring the WEKA REST API.

Performance: Explore the results of FIO performance tests on the WEKA filesystem, ensuring optimal system performance.

WEKA filesystems & object stores: Understand the role and management of filesystems, object stores, filesystem groups, and key-management systems within WEKA configurations.

Additional protocols: Learn about the supported protocols—NFS, SMB, and S3—for accessing data stored in a WEKA filesystem.

Operation guide: Navigate through various WEKA system operations, including events, statistics, user management, upgrades, expansion, and more.

Billing & licensing: Gain insights into WEKA system licensing options and alternative billing approaches.

Monitor the WEKA cluster: Effectively monitor your WEKA cluster by deploying the WEKA Management Server (WMS) alongside tools like Local WEKA Home, WEKAmon, and SnapTool.

WEKA support: Find guidance on obtaining support for the WEKA system and effectively managing diagnostics.

Best practice guides: Explore our carefully selected guides, starting with WEKA and Slurm integration, to discover expert-recommended strategies and insights for optimizing your WEKA system and achieving peak performance in various scenarios.

Conventions

The documentation marks the CLI mandatory parameters with an asterisk (*).

Documentation feedback

We welcome your feedback to improve our documentation. Include the document version and topic title with your suggestions and email them to . For technical inquiries, contact our . Thank you for helping us maintain high-quality resources.

Redundancy in WEKA system deployments can vary, ranging from 3+2 to 16+4. Choosing the most suitable configuration involves several key considerations, including redundancy levels, data stripe width, hot spare capacity, and the performance required during data rebuilds.

Redundancy levels

Redundancy can be configured as N+2 or N+4, directly impacting capacity and performance. A redundancy level of 2 is typically sufficient for most configurations, while redundancy levels of 4 are reserved for larger clusters with 100 or more backends or critical data scenarios.

The WEKA filesystem (WekaFS™) redefines storage solutions with its software-only approach, compatible with standard AMD or Intel x86-based servers and NVMe SSDs. It eliminates the need for specialized hardware, allowing easy integration of technological advancements without disruptive upgrades. WekaFS addresses common storage challenges by removing performance bottlenecks, making it suitable for environments requiring low latency, high performance, and cloud scalability.

Use cases span various sectors, including AI/ML, Life Sciences, Financial Trading, Engineering DevOps, EDA, Media Rendering, HPC, and GPU pipeline acceleration. Combining existing technologies and engineering innovations, WekaFS delivers a powerful, unified solution that outperforms traditional storage systems, efficiently supporting various workloads.

WekaFS is a fully distributed parallel filesystem leveraging NVMe Flash for file services. Integrated tiering seamlessly expands the namespace to and from HDD object storage, simplifying data management. The intuitive GUI allows easy administration of exabytes of data without specialized storage training.

WekaFS stands out with its unique architecture, overcoming legacy systems’ scaling and file-sharing limitations. Supporting POSIX, NFS, SMB, S3, and GPUDirect Storage, it offers a rich enterprise feature set, including snapshots, clones, tiering, cloud-bursting, and more.

Benefits include high performance across all IO profiles, scalable capacity, robust security, hybrid cloud support, private/public cloud backup, and cost-effective flash-disk combination. WekaFS ensures a cloud-like experience, seamlessly transitioning between on-premises and cloud environments.

WekaFS functionality running in its RTOS within the Linux container (LXC) is comprised of the following software components:

• File services (frontend): Manages multi-protocol connectivity.

• File system computing and clustering (backend): Manages data distribution, data protection, and file system metadata services.

• SSD drive agent: Transforms the SSD into an efficient networked device.

By bypassing the kernel, WekaFS achieves faster, lower-latency performance, portable across bare-metal, VM, containerized, and cloud environments. Efficient resource consumption minimizes latency and optimizes CPU usage, offering flexibility in shared or dedicated environments.

WekaFS design departs from traditional NAS solutions, introducing multiple filesystems within a global namespace that share the same physical resources. Each filesystem has its unique identity, allowing customization of snapshot policies, tiering, role-based access control (RBAC), quota management, and more. Unlike other solutions, filesystem capacity adjustments are dynamic, enhancing scalability without disrupting I/O.

The WEKA system offers a robust, distributed, and highly scalable storage solution, allowing multiple application servers to access shared filesystems efficiently and with solid consistency and POSIX compliance.

Related information

Raw capacity

Raw capacity is the total capacity on all the SSDs assigned to a WEKA system cluster. For example, 10 SSDs of one terabyte each have a total raw capacity of 10 terabytes. This is the total capacity available for the WEKA system. This will change automatically if more servers or SSDs are added.

The WEKA system offers a range of powerful functionalities designed to enhance data protection, scalability, and efficiency, making it a versatile solution for various storage requirements.

Protection

The WEKA system employs N+2 or N+4 protection, ensuring data protection even in the face of concurrent drive or backend failures and maintaining the WEKA system up and running to provide continuous services. This complex protection scheme is determined during cluster formation and can vary, offering configurations starting from 3+2 up to 16+2 for larger clusters.

Distributed network scheme

The WEKA system incorporates an any-to-any protection scheme that ensures the rapid recovery of data in the event of a backend failure. Unlike traditional storage architectures, where redundancy is often established across backend servers (backends), WEKA's approach leverages groups of datasets to protect one another within the entire cluster of backends.

Here's how it works:

• Data recovery process: If a backend within the cluster experiences a failure, the WEKA system initiates a rebuilding process using all the other operational backends. These healthy backends work collaboratively to recreate the data that originally resided on the failed backend. Importantly, all this occurs in parallel, with multiple backends simultaneously reading and writing data.

• Speed of rebuild: This approach results in a speedy rebuild process. In a traditional storage setup, only a small subset of backends or drives actively participate in rebuilding, often leading to slow recovery. In contrast, in the WEKA system, all but the failed backend are actively involved, ensuring swift recovery and minimal downtime.

• Scalability benefits: The advantages of this distributed network scheme become even more apparent as the cluster size grows. In larger clusters, the rebuild process is further accelerated, making the WEKA system an ideal choice for organizations that need to handle substantial data volumes without sacrificing data availability.

In summary, the WEKA system's distributed network scheme transforms data recovery by involving all available backends in the rebuild process, ensuring speedy and efficient recovery, and this efficiency scales with larger clusters, making it a robust and scalable solution for data storage and protection.

Efficient component replacement

In the WEKA system, a hot spare is configured within the cluster to provide the additional capacity needed for a full recovery after a rebuild across the entire cluster. This differs from traditional approaches, where specific physical components are designated hot spares. For instance, in a 100-backend cluster, sufficient capacity is allocated to rebuild the data and restore full redundancy even after two failures. The system can withstand two additional failures depending on the protection policy and cluster size.

This strategy for replacing failed components does not compromise system vulnerability. In the event of a system failure, there's no immediate need to physically replace a failed component with a functional one to recreate the data. Instead, data is promptly regenerated, while replacing the failed component with a working one is a background process.

Enhanced fault tolerance with failure domains

In the WEKA system, failure domains are groups of backends that could fail due to a single underlying issue. For instance, if all servers within a rack rely on a single power circuit or connect through a single ToR switch, that entire rack can be considered a failure domain. Imagine a scenario with ten racks, each containing five WEKA backends, resulting in a cluster of 50 backends.

To enhance fault tolerance, you can configure a protection scheme, such as 6+2 protection, during the cluster setup. This makes the WEKA system aware of these possible failure domains and creates a protection stripe across the racks. This means the 6+2 stripe is distributed across different racks, ensuring that the system remains operational even in case of a complete rack failure, preventing data loss.

It's important to note that the stripe width must be less than or equal to the count of failure domains. For instance, if there are ten racks, and one rack represents a single point of failure, having a 16+4 cluster protection is not feasible. Therefore, the level of protection and support for failure domains depends on the stripe width and the chosen protection scheme.

Prioritized data rebuild process

Seamless distribution, scaling, and enhanced performance

In the WEKA system, every client installed on an application server directly connects to the relevant WEKA backends that store the required data. There's no intermediary backend that forwards access requests. Each WEKA client maintains a synchronized map, specifying which backend holds specific data types, creating a unified configuration shared by all clients and backends.

When a WEKA client attempts to access a particular file or offset in a file, a cryptographic hash function guides it to the appropriate backend containing the needed data. This unique mechanism enables the WEKA system to achieve linear performance growth. It synchronizes scaling size with scaling performance, providing remarkable efficiency.

For instance, when new backends are added to double the cluster's size, the system instantly redistributes part of the filesystem data between the backends, resulting in an immediate double performance increase. Complete data redistribution is unnecessary even in modest cluster growths, such as moving from 100 to 110 backends. Only a fraction (10% in this example) of the existing data is copied to the new backends, ensuring a balanced distribution and active participation of all backends in read operations.

The speed of these seamless operations depends on the capacity of the root backends and network bandwidth. Importantly, ongoing operations remain unaffected, and the system's performance improves as data redistribution occurs. The finalization of the redistribution process optimizes both capacity and performance, making the WEKA system an ideal choice for scalable and high-performance storage solutions.

Efficient data reduction

WEKA offers a cluster-wide data reduction feature that can be activated for individual filesystems. This capability employs block-variable differential compression and advanced de-duplication techniques across all filesystems to significantly reduce the storage capacity required for user data, resulting in substantial cost savings for customers.

The effectiveness of the compression ratio depends on the specific workload. It is particularly efficient when applied to text-based data, large-scale unstructured datasets, log analysis, databases, code repositories, and sensor data. For more information, refer to the dedicated topic.

Many hardware vendors ship their products with the SR-IOV feature disabled. The feature must be enabled on such platforms before installing the Weka system. Enabling the SR-IOV applies to the server BIOS.

If the SR-IOV is already enabled, it is recommended to verify the current state before proceeding with the installation of the Weka system.

Before you begin

Verify that the NIC drivers are installed and loaded successfully. If it still needs to be done, perform the Install NIC drivers procedure.

Enable SR-IOV in the server BIOS

The following procedure is a vendor-specific example and is provided as a courtesy. Depending on the vendor, the same settings may appear differently or be located elsewhere. Therefore, refer to your hardware platform and NIC vendor documentation for the latest information and updates.

The WEKA client

The WEKA client is a standard POSIX-compliant filesystem driver installed on application servers, facilitating file access to WEKA filesystems. Acting as a conventional filesystem driver, it intercepts and executes all filesystem operations, providing applications with local filesystem semantics and performance—distinct from NFS mounts. This approach ensures centrally managed, shareable, and resilient storage for WEKA.

Tightly integrated with the Linux Page Cache, the WEKA client leverages this transparent caching mechanism to store portions of filesystem content in the client's RAM. The Linux operating system maintains a page cache in the unused RAM, allowing rapid access to cached pages and yielding overall performance enhancements.

The Linux Page Cache, implemented in the Linux kernel, operates transparently to applications. Utilizing unused RAM capacity, it incurs minimal performance penalties, often appearing as "free" or "available" memory.

WEKA on GCP overview

Leveraging GCP's advantages, WEKA offers a customizable terraform-gcp-weka module for deploying the WEKA cluster on GCP. In GCP, WEKA operates on instances, each capable of using up to eight partitions of drives on the connected physical server (without direct drive usage). These drives can be shared among partitions for other clients on the same server.

WEKA requires a minimum of four VPC networks, each associated with one of the instances. This configuration aligns with the four key WEKA processes: Compute, Drive, Frontend, and Management, with each process requiring a dedicated network interface as follows:

Object stores in WEKA serve as optional external storage media, complementing SSD storage with a more cost-effective solution. This allows for the strategic allocation of resources, with object stores accommodating warm data (infrequently accessed) and SSDs handling hot data (frequently accessed).

In WEKA, object store buckets can be distributed across different physical object stores. However, to ensure optimal Quality of Service (QoS), a crucial mapping between the bucket and the physical object store is required.

WEKA treats object stores as physical entities, either on-premises or in the cloud, grouping multiple object store buckets. These buckets can be categorized as either local (used for tiering and snapshots) or remote (exclusively for snapshots). An object-store bucket must be added to an object store with the same type and remain inaccessible to other applications.

While a single object store bucket can potentially serve different filesystems and multiple WEKA systems, it is advisable to dedicate each bucket to a specific filesystem. For instance, if managing three tiered file systems, assigning a dedicated local object storage bucket to each file system is recommended.

For each filesystem, users can attach up to three object store buckets:

WEKA provides a ready-to-deploy Terraform package for installing the WEKA cluster on AWS Virtual Private Cloud (VPC).

The following diagram provides an overview of the various steps automated with the Terraform-driven provisioning of the WEKA cluster backend servers on AWS EC2 instances.

Workflow description

Deployment prerequisites

• Check that your AWS account limits allow for the deployment of your selected configuration. You can check your limits under the Limits tab in the EC2 console.

Once the Terraform modules are applied, two workflows are running every minute. One for scale-up and the other for scale-down.

WEKA provides a cloud function for scale-up or scale-down of the number of compute engine instances (cluster size). Terraform automatically creates the cluster according to the specified target value in a few minutes.

To change the cluster size (up or down), specify the link to the resize cloud function on GCP and the resize target value for the number of compute engine instances in the following command and run it:

Example:

Related topics

The management of filesystems is an integral part of the successful running and performance of the WEKA system and overall data lifecycle management.

Related topics

The transition from SSD-only filesystem to tiered filesystem

An SSD-only filesystem can be reconfigured as a tiered filesystem by attaching an object store. In such a situation, the default is to maintain the filesystem size. In order to increase the filesystem size, the total capacity field can be modified, while the existing SSD capacity remains the same.

A filesystem group in the WEKA system is used specifically to manage tiering policies for filesystems. It defines key parameters, including the drive retention period and the tiering queue time, which determine how and when data is tiered.

When you add a filesystem, it must be associated with a filesystem group to apply these tiering behaviors. The WEKA system supports up to eight filesystem groups, allowing flexibility in managing tiering policies across different filesystems.

Related topics

The GCP Console has a interface in which you can view the cloud function logs related to the WEKA cluster activities, such as when scaling instances up or down. In addition, the cluster state file retained in the cloud storage provides you with the status of the operations in the WEKA project.

Typical troubleshooting flow if the resize cloud function does not resize the cluster

1. Open the cluster state file and check that the desired_size is as expected and the clusterized value is true. The cluster state file is in the cloud storage, and its name comprises the prefix and cluster_name

This section explains how data lifecycle is maintained when working with a tiered WEKA system configuration, together with the options for control. The following subjects are covered:

• Advanced explanation of .

• System behavior when .

WEKA supports client instances with at least two NICs, one for management and one for the frontend data. It is possible to add more NICs for redundancy and higher performance.

A client with the same VPC networks and subnets as the cluster can connect without additional configuration. If a client is on another VPC network, peering is required between the VPC networks.

The client instance must be in the same region as the WEKA cluster on GCP.

Mount the filesystem

1. Create a mount point (only once):

1. Install the WEKA agent (only once):

Example:

1. Mount a stateless client on the filesystem. In the mount command, specify all the NICs of the client.

• DPDK mount with four NICs:

Example:

• UDP mount:

Example:

Related topics

The Terraform-AWS-WEKA module is an open-source repository. It contains modules to customize the WEKA cluster installation on AWS. The default protocol deployed using the module is POSIX.

The Terraform-AWS-WEKA module supports public and private cloud deployments. All deployment types require passing the get.weka.io token to Terraform to download the WEKA release from the public get.weka.io service.

Terraform-AWS-WEKA module components

The Terraform-AWS-WEKA module consists of the following components:

• Required module:

  • WEKA Root Module is located in the main Terraform module.

• Optional sub-modules:

Terraform-AWS-WEKA example

The following is a basic example in which you provide the minimum detail of your cluster, and the Terraform module completes the remaining required resources, such as VPC, subnets, security group, placement group, DNS zone, and IAM roles.

You can use this example as a reference to create the main.tf file.

Attachment of a local object store bucket to a filesystem

Two local object store buckets can be attached to a filesystem, but only one of the buckets is writable. A local object store bucket is used for both tiering and snapshots. When attaching a new local object store bucket to an already tiered filesystem, the existing local object store bucket becomes read-only, and the new object store bucket is read/write. Multiple local object stores allow a range of use cases, including migration to different object stores, scaling of object store capacity, and increasing the total tiering capacity of filesystems.

When attaching a local object store bucket to a non-tiered filesystem, the filesystem becomes tiered.

Detachment of a local object store bucket from a filesystem

Detaching a local object store bucket from a filesystem migrates the filesystem data residing in the object store bucket to the writable object store bucket (if one exists) or to the SSD.

When detaching, the background task of detaching the object store bucket begins. Detaching can be a long process, depending on the amount of data and the load on the object stores.

• Migration to a different object store: When detaching from a filesystem tiered to two local object store buckets, only the read-only object store bucket can be detached. In such cases, the background task copies the relevant data to the writable object store. In addition, the allocated SSD capacity only requires enough SSD capacity for the metadata.

• Un-tiering a filesystem: Detaching from a filesystem tiered to one object store bucket un-tiers the filesystem and copies the data back to the SSD. The allocated SSD capacity must be at least the total capacity the filesystem uses.

On completion of detaching, the object store bucket does not appear under the filesystem when using the weka fs command. However, it still appears under the object store and can be removed if any other filesystem does not use it. The data in the read-only object store bucket remains in the object store bucket for backup purposes. If this is unnecessary or the reclamation of object store space is required, it is possible to delete the object store bucket.

Migration considerations

When migrating data (using the detach operation), copy only the necessary data (to reduce migration time and capacity). However, you may want to keep snapshots in the old object store bucket.

Migration workflow

The order of the following steps is important.

1. Attach a new object store bucket (the old object store bucket becomes read-only).

2. Delete any snapshot that does not need to be migrated. This action keeps the snapshot on the old bucket but does not migrate its data to the new bucket.

3. Detach the old object store bucket.

Attach a remote object store bucket

One remote object store bucket can be attached to a filesystem. A remote object store bucket is used for backup. Only snapshots are uploaded using Snap-To-Object. The snapshot uploads are incremental to the previous one.

Detach a remote object store bucket

Detaching a remote object store bucket from a filesystem keeps the backup data within the bucket intact. It is still possible to use these snapshots for recovery.

Related topics

The WEKA project ultimately uses the internal GCP resources. A basic WEKA project includes a cluster with several virtual private clouds (VPCs), VMs (instances), a load balancer, DNS, cloud storage, a secret manager, and a few more elements that manage the resize of the cluster. The peering between all the virtual networks enables running the functions across all the networks.

Resize cloud function operation

A resize cloud function in vpc-0 and a workload listener are deployed for auto-scale instances in GCP. Once a user sends a request for resizing the number of instances in the cluster, the workload listener checks the cluster state file in the cloud storage and triggers the resize cloud function if a resize is required. The cluster state file is an essential part of the resizing decision. It indicates states such as:

• Readiness of the cluster.

• The number of existing instances.

• The number of requested instances.

The secret manager retains the user name (usually admin) and the Terraform-generated password. The resize cloud function uses the user name and password to operate on the cluster instances.

GCP subnet types

Depending on the required security level, you can deploy the WEKA project on one of the following subnet types:

• Public subnet: Use a public subnet within your VPC with an internet gateway, and allow public IP addresses for your instances.

• Private subnet shared with a Bastion project: Create a private subnet with a shared project with a Bastion project, a risk-based security solution used for authenticating communication with a public network, such as downloading the WEKA software from get.weka.io. The Bastion project includes a Bastion VM (host) acting as a network gateway. The relevant ports are open (by the Terraform files).

• Private subnet shared with a yum project: If a connection to get.weka.io for downloading the WEKA software is impossible, create a private subnet with a yum repository containing the WEKA software. The relevant ports are open (by the Terraform files).

Auto-scaling is useful to easily scale the number of EC2 instances up or down at need.

After deploying the WEKA cluster via CloudFormation, it is possible to create an auto-scaling group to ease the WEKA cluster size management.

You can create an auto-scaling group for your cluster by running the wekactl utility.

You can control the number of instances by either changing the desired capacity of instances from the AWS auto-scaling group console or defining your custom metrics and scaling policy in AWS. Once the desired capacity has changed, WEKA takes care of safely scaling the instances.

For more information and documentation on the utility, refer to the .

Overview

Managing authentication across multiple clusters in the WEKA CLI is streamlined with connection profiles. By default, when you run the weka user login command, it creates a profile stored as .weka/auth-token.json. This is sufficient for single-cluster environments. However, in a multi-cluster environment, use the --profile parameter to create and manage separate profiles for each cluster. This allows you to switch between clusters without needing to re-authenticate each time, enhancing efficiency and usability.

Profile naming conventions

When creating a connection profile, follow these guidelines:

• Maximum length: 50 characters

• Allowed characters:

  • Alphanumeric (A-Z, a-z, 0-9)

  • Underscores (_)

Profile names dictate where authentication details are stored in the .weka directory:

• Default profile: .weka/auth-token.json

• Named profiles: .weka/auth-token-<profile-name>.json

Log in with profiles

The weka user login command supports profiles, enabling you to specify which profile to use or create a new one.

Command syntax:

• Default profile: If no profile is specified, the system uses the default profile.

• Profile-specific file: Authentication information is saved in a file named after the profile.

• Success message: After a successful login, the following message appears:

• Failure message: If the profile is not found or the login fails, an error message displays the profile name and file path.

Log out of profiles

The weka user logout command supports profiles, enabling you to remove the authentication details for a specific profile.

Command syntax:

• The specified profile’s authentication file is deleted.

• If no profile is specified, the default profile is logged out.

Using profiles with WEKA CLI commands

You can specify a profile when executing most WEKA CLI commands using the --profile option. If no profile is provided, the default profile is used.

Command syntax:

Related topic

Scale-out the WEKA cluster backend servers

Scale-out is the process of increasing the number of EC2 instances in the system to handle higher workloads or enhance redundancy.

Scale-out is essential to ensure a system can meet growing demands, maintain performance, and distribute workloads effectively. This proactive approach helps prevent overloads, reduce response times, and maintain high availability.

Action

• Increase the desired size of the Auto-Scaling Group (ASG) associated with your WEKA cluster using the AWS Console or AWS CLI.

Result

• AWS automatically launches the new EC2 instance.

• AWS triggers the Lambda Function to create a join script that runs once as part of the instance user data and, subsequently, integrates the new EC2 instance into the existing WEKA cluster.

You can monitor the process in the AWS Step Function GUI.

Scale-in the WEKA cluster backend servers

Scale-in is the process of reducing the number of EC2 instances of a system to align with decreased workloads or to optimize resource utilization.

Scale-in is essential for efficient resource management, cost reduction, and ensuring the appropriate allocation of resources. It helps prevent over-provisioning, lowers operational expenses, and safeguards against unintentional removal of EC2 instances from the existing WEKA cluster in AWS.

The cluster is configured with scale-in protection and instance termination protection to enhance the safety of this process.

Action

• Decrease the desired size of the Auto-Scaling Group (ASG) associated with your WEKA cluster. You can do this through the AWS Console, AWS CLI, or other compatible methods.

Result

• After modifying the desired size, it doesn't immediately impact the Auto Scaling Group (ASG). Instead, a Step Function continuously monitors the configuration.

• This Step Function runs every minute and identifies that the desired size is less than the current WEKA system's size.

• When this condition is met, it initiates a scale-in process, but only if certain conditions are met, such as having enough capacity on the filesystem.

If you are not using the WMS or WSA automated tools for installing a WEKA cluster, manually install a supported OS and the WEKA software on the bare metal server.

Procedure

1. Follow the relevant Linux documentation to install the operating system, including the required packages.

Required packages

Using the GUI, you can:

Before a WEKA system can use a Broadcom BCM57508-P2100G, the server must have the necessary drivers and firmware from Broadcom's download center.

Set up the drivers and software

Procedure:

1. Download software bundle: Access Broadcom's download center and download the software bundle onto the target server. Carefully review the instructions included in the bundle.

AWS

AWS cluster

Snapshots allow the saving of a filesystem state to a .snapshots directory under the root filesystem. They can be used for:

• Physical backup: The snapshots directory can be copied into a different storage system, possibly on another site, using either the WEKA system Snap-To-Object feature or third-party software.

• Logical backup: Periodic snapshots enable filesystem restoration to a previous state if logical data corruption occurs.

Overview

There are several levels on the WEKA system where capacity usage can be restricted.

1. On an organization level: Set a different organization to manage its filesystems, where quotas for an organization can be set, as described in the section.

This guide outlines the customization process for Terraform configurations to deploy the WEKA cluster on AWS. It is designed for system engineers with expertise in AWS and Terraform. Start by creating a main.tf file and adapting it to your AWS deployment requirements. Once configured to your preferences, proceed to apply the changes.

Create a main.tf file

Before you begin

The must be installed on the workstation used for the deployment. Check the minimum required Terraform version specified in the section of the Terraform-AWS-WEKA module.

Procedure

1. Review the and use it as a reference for creating the main.tf according to your deployment specifics on AWS.

2. Tailor the main.tf file to create SMB-W or NFS protocol clusters by adding the relevant code snippet. Adjust parameters like the number of gateways, instance types, domain name, and share naming:

• SMB-W

• NFS

1. Add WEKA POSIX clients (optional): If needed, add to support your workload by incorporating the specified variables into the main.tf file:

Apply the main.tf file

Once you complete the main.tf settings, apply it: Run terraform apply

Create a dedicated Cluster Admin username and password

When deploying a WEKA cluster on the cloud using Terraform, a default username (admin) is automatically generated, and Terraform creates the password. Both the username and password are stored in the AWS Secrets Manager. This user facilitates communication between the cloud and the WEKA cluster, particularly during scale-up and scale-down operations.

As a best practice, it’s recommended to create a dedicated local user in the WEKA cluster with the Cluster Admin role. This user will serve as a service account for cloud-cluster communications.

Procedure

1. Create a local user with the Cluster Admin role in the WEKA cluster.

2. In the AWS Secrets Manager, navigate to Secrets.

3. Update the weka_username and weka_password services with the username and password of the newly created local user.

Related topic

Register to get.weka.io

To sign in to get.weka.io, you first need to create an account and fill in your details. If you already have a registered account for get.weka.io, skip this procedure.

Procedure

1. Go to the download site, and select Create an account.

The Send Registration Email page opens.

2. Fill in your organization's email address (private mail is prohibited). Select I’m not a robot, and then select Send Registration Email.

3. Check your inbox for a registration email from Weka.io. To confirm your registration, select the link. The Create Your Account page opens.

4. Fill in your email address, full name, and password. Then, select Create Account.

Your request for access to is sent to WEKA for review. Wait for a validation email. Once your registration is approved, you can sign in to .

Download the WEKA installation packages

Download the required WEKA installation packages according to the workflow path.

• Path A (automated with WMS and WSA): Download the WMS and WSA ISOs from . The WMS is downloaded from a dedicated dropdown. The WSA is found in the relevant release page.

• Path B (automated with WSA): Download the WSA package from The WSA is found in the relevant release page.

• Path C (manual installation and configuration): Download the WEKA software tarball from . The tarball is found in the relevant release page.

You can only sign in and download the packages if you are a registered user.

Procedure: Download from get.weka.io

1. Go to the download site, and sign in with your registered account.

page opens.

1. Do one of the following:

   • Select the required package from the dashboard.

   • Select the Releases tab, select the required release, and follow the download instructions. (The token in the download link is purposely blurred.)

What to do next?

Depending on the workflow path you follow, go to one of the following:

(path A)

(path B)

(path C)

Using the GUI, you can perform the following actions:

• View filesystem groups

• Add filesystem groups

• Edit filesystem groups

View filesystem groups

The filesystem groups are displayed on the Filesystems page. Each filesystem group indicates the number of filesystems that use it.

Procedure

1. From the menu, select Manage > Filesystems.

Add a filesystem group

A filesystem group is required when adding a filesystem. You can create more filesystem groups if you want to apply a different tiering policy on specific filesystems.

Procedure

1. From the menu, select Manage > Filesystems.

2. Select the + sign right to the Filesystem Groups title.

3. In the Create Filesystem Group dialog, set the following:

1. Select Create.

Related topics

To learn more about the drive retention period and tiering cue, see:

Edit a filesystem group

You can edit the filesystem group policy according to your system requirements.

Procedure

1. From the menu, select Manage > Filesystems.

2. Select the filesystem group you want to edit.

3. Select the pencil sign right to the filesystem group name.

4. In the Edit Filesystem Group dialog, update the settings as you need. (See the parameter descriptions in the

1. Select Update.

Delete a filesystem group

You can delete a filesystem group no longer used by any filesystem.

Procedure

1. From the menu, select Manage > Filesystems.

2. Select the filesystem group you want to delete.

3. Verify that the filesystem group is not used by any filesystems (indicates 0 filesystems).

1. Select the Remove icon. In the pop-up message, select Yes to delete the filesystem group.

Overview

Mounting filesystems from a single WEKA client to multiple clusters provides several advantages:

• Expanded cluster connectivity: A single client can connect to up to seven clusters simultaneously, increasing storage capacity and computational capabilities.

• Unified data access: Provides a consolidated view of data across multiple clusters, simplifying access and management while improving data availability, flexibility, and resource efficiency.

• Optimized workload distribution: Enables efficient workload distribution across clusters, supporting scalable applications and enhancing overall performance.

• Seamless integration: WEKA’s SCMC feature ensures smooth and efficient integration for clients accessing multiple clusters.

Bandwidth division considerations in SCMC

The bandwidth division in SCMC is a universal consideration based on the specific NIC's bandwidth. It applies across various NIC types, including those using DPDK or specific models like the X500-T1.

During SCMC mounts, each active connection can use the bandwidth available on its associated NIC port. This is true during peak usage and idle cases. In scenarios where NICs are dual-ported, each connection operates independently, leveraging its dedicated port.

When working with low-bandwidth NICs such as the X500-T1, a 10Gb/s NIC, consider bandwidth calculations. In the context of SCMC, each container (representing connectivity to a different cluster) uses half of the available bandwidth (5Gb/s) for a shared port. Note that a dual-port NIC has a dedicated port for each container, optimizing bandwidth distribution. Keep these factors in mind for an optimal SCMC setup.

When a stateless client mounts a filesystem in a cluster, it creates a client container with the same version as provided by the cluster. Because there may be situations where some of the clusters run a different WEKA version than the others, such as during an upgrade, it is required to set the same client target version on all clusters. The client target version is retained regardless of the cluster upgrade.

Procedure:

1. Connect to each cluster and run the following command to set the client target version.

Where: <version> is the designated client target version, which will be installed on the client container upon the mount command. Ensure this version is installed on the backend servers.

1. To display the existing client target version in the cluster, run the following command:

1. To reset the client target version to the cluster version, run the following command:

Mount a stateless client container on multiple clusters

Use the same commands as with a single client.

To mount a stateless client using UDP mode, add -o net=udp -o core=<core-id> to the command line. For example:

Mount persistent client containers on multiple clusters

For persistent client containers, the client-target-version parameter is not relevant. The version of the client container is determined when creating the container in the WEKA client using the weka local setup container command. Therefore, ensure that all client containers in the WEKA client have the same minor version as in the clusters.

To mount a persistent client container to a cluster, specify the container name for that mount.

Run commands from a server with multiple client containers

When running WEKA CLI commands from a server hosting multiple client containers, each connected to a different WEKA cluster, it’s required to specify the client container port or the backend IP address/name of the cluster (linked to that client) in the command.

Consider a server with two client containers:

To run a WEKA CLI command on the second cluster (associated with client2), use either of the following methods:

• By specifying the backend IP address or name linked to that client container (assuming the backend name is DataSphere2-1):

• By specifying the client container port:

This approach ensures that your WEKA CLI command targets the correct WEKA cluster associated with the specified client container.

Related topics

Required services

The AWS region must support the following services used in WEKA on AWS.

• AWS EC2 Instances

• AWS Lambda

• AWS Step Functions

• AWS CloudWatch event rule

• AWS S3 (required for snap/tiering to object)

Supported regions

Related topic

Related information

Using the CLI, you can perform the following actions:

• View filesystem groups

• Add filesystem groups

• Edit filesystem groups

View filesystem groups

Command: weka fs group

Use this command to view information on the filesystem groups in the WEKA system.

Add a filesystem group

Command: weka fs group create

Use the following command to add a filesystem group:

weka fs group create <name> [--target-ssd-retention=<target-ssd-retention>] [--start-demote=<start-demote>]

Parameters

Edit a filesystem group

Command: weka fs group update

Use the following command to edit a filesystem group:

weka fs group update <name> [--new-name=<new-name>] [--target-ssd-retention=<target-ssd-retention>] [--start-demote=<start-demote>]

Parameters

Delete a filesystem group

Command: weka fs group delete

Use the following command line to delete a filesystem group:

weka fs group delete <name>

Parameters

Related topics

To learn about the tiring policy, see:

Cluster architecture basics

In the WEKA system, servers operate as members of a cluster, with each server hosting multiple containers. These containers run software instances, referred to as processes, that collaborate and communicate within the cluster to deliver robust and efficient storage services. This architecture ensures scalability and fault tolerance by distributing storage functionality across interconnected containers.

Process types and core requirements

The WEKA system uses different types of processes, each dedicated to specific functions:

• Drive processes: Manage SSD drives and handle IO operations to drives. These processes are fundamental to storage operations and each requires a dedicated core to ensure optimal performance.

• Compute processes: Handle filesystems, cluster-level functions, and IO from clients. The dedicated core requirement for each compute process ensures consistent processing power for these critical operations.

• Frontend processes: Also known as client processes, manage POSIX client access and coordinate IO operations with compute and drive processes. Each frontend process needs a dedicated core to maintain responsive client interactions.

Multi-Container Backend architecture (MCB)

In the WEKA cluster, each server implements a multi-container backend architecture where containers are specialized by process type (drive, compute, or frontend).

Benefits of MCB architecture

• Non-Disruptive Upgrade (NDU) capabilities:

  • Enables true non-disruptive upgrades where containers can run different software versions independently without system interruption

  • Supports individual container rollback without impacting cluster operations

  • Maintains continuous network control plane access throughout the upgrade process, ensuring uninterrupted client service

System limitations and specifications

Process limits

• Total processes per cluster: 65,534 (includes all process types: management, drive, compute, and frontend)

• Maximum backend processes: 25,000 (excludes frontend processes)

• Maximum management processes: 32,767

• Maximum drive processes: 62,244

Server and container limits

• Maximum WEKA cores per server: 64

• Maximum cores per container: 19

The WEKA® Data Platform on AWS provides a fast and scalable platform for running performance-intensive applications and hybrid cloud workflows.

WEKA provides a ready-to-deploy Terraform package that you can customize for installing the WEKA cluster on AWS. Optionally, you can install the WEKA cluster using the AWS CloudFormation.

Ensure you are familiar with the following concepts and services that are used for the WEKA installation on AWS:

Supported machine types for backends

The following table provides the supported machine types (VM instance) for backends (and clients) applied by the Terraform package:

WEKA provides a ready-to-deploy that you can customize to install the WEKA cluster on GCP.

The Terraform package contains the following modules:

• setup_network: includes vpcs, subnets, peering, firewall, and health check.

• service_account: includes the service account used for deployment with all necessary permissions.

The WEKA system provides a RESTful API, enabling you to automate interactions with the WEKA system and integrate them into your workflows or monitoring systems.

Access the REST API

NFS (Network File System) is a protocol that enables clients to access the WEKA filesystem without requiring WEKA's client software. This leverages the standard NFS implementation of the client's operating system.

WEKA supports an advanced NFS implementation, NFS-W, designed to overcome inherent limitations in the NFS protocol. NFS-W is compatible with NFSv3 and NFSv4 protocols, offering enhanced capabilities, including support for more than 16 user security groups.

The legacy NFS stack is also available for backward compatibility, supporting only the NFSv3 protocol and a maximum of 16 user security groups.

Directory quotas monitor the filesystem capacity usage by a directory and allow restricting the amount of space used by the directory.

Using the GUI, you can:

Using CloudFormation deployment saves a lot of potential errors that may occur during installation, especially in the configuration of security groups and other connectivity-related issues. However, the following errors related to the following subjects may occur during installation:

• Installation logs

• AWS account limits

Using the CLI, you can:

WEKA provides a variety of tools for automating the WEKA software installation process. These include:

• WEKA Management Station (WMS)

• WEKA Software Appliance (WSA)

• WEKA Configurator

WEKA Management Station (WMS)

WMS can be used to speed the WEKA Software Appliance (WSA) deployment on the supported bare metal servers: Dell, HPE, Lenovo, and Supermicro.

This is the preferred installation method, the simplest and fastest method to get from bare metal to a working WEKA cluster. If you cannot meet the prerequisites for deploying WMS, use the WSA package.

WEKA Software Appliance (WSA)

WSA is a WEKA server image deployed with a preconfigured operating system. This method significantly speeds up the OS and WEKA cluster installation and provides a WEKA-supported operating environment.

After installation, the server is in STEM mode, which is the initial state before the configuration.

If you cannot use the WSA for WEKA cluster installation, review the requirements and follow the instructions for deploying WEKA using the WEKA Configurator.

WEKA Configurator

The WEKA Configurator automatically generates the WEKA Cluster configurations (config.sh) to apply on the cluster servers.

High-level deployment workflow

The following illustrates a high-level deployment workflow on a group of bare metal servers.

Deployment workflow paths summary

The following summarizes the three workflow paths to install and configure the WEKA cluster.

• Path A: Automated with WMS and WSA

• Path B: Automated with WSA only

• Path C: Manual installation and configuration

Select the path applicable to your needs.

What do do next?

(all paths)

During the deployment of the WEKA system, the EC2 instances require access to the internet to download the WEKA software. For this reason, you need to deploy the WEKA system in one of the following deployment types in AWS:

• Public subnet: Use a public subnet within your VPC with an internet gateway, and allow public IP addresses for your instances.

• Private subnet with NAT Gateway: Create a private subnet with a route to a NAT gateway with an elastic IP in the public subnet.

• Private subnet using WEKA VPC endpoint: Requires the creation of a (once per VPC) that creates the necessary resources.

• Private subnet using custom proxy: Requires the creation of a (once per VPC) that creates the necessary resources.

The following diagrams illustrate the components of the public subnet and private subnet with NAT gateway deployment types in AWS.

Update the number of vCPU limits in EC2

By default, AWS does not provide enough vCPUs to install a WEKA system. Use the Limits Calculator for your region from the AWS EC2 dashboard.

Procedure

1. On the AWS EC2 dashboard, select the Limits option from the left menu.

2. In the Limits Calculator, do the following:

• In the Current Limit, set the number of vCPUs you currently have for a region.

• In the vCPUs needed, set the required number of vCPUs for your specific deployment.

Select the Request on-demand limit increase link to get more vCPUs.

The following example shows the required vCPUs for a six servers cluster with two clients of type i3en.2xlarge. This example is the smallest type of instance for a WEKA system deployment.

After the installation on AWS best practices

Backup and recovery

Resiliency

The Weka system is a distributed cluster protected from 2 or 4 failure domain failures, providing fast rebuild times. For details, see the section.

Instance failure

If an instance failure occurs, the Weka system rebuilds the data. Add a new instance to the cluster to regain the reduced compute and storage due to the instance failure.

Upload snapshots to S3

It is advisable to use periodic (incremental) snapshots to back up the data and protect it from multiple EC2 instances failures.

The recovery point objective (RPO) is determined by the cadence in which the snapshots are taken and uploaded to S3. The RPO changes between the type of data, regulations, and company policies, but it is advisable to upload at least daily snapshots of the critical filesystems. For details, see the section.

If a failure occurs and it is required to recover from a backup, spin up a cluster using the or , and create filesystems from those snapshots. You do not need to wait for the data to reach the EC2 volumes. It is instantly accessible through S3. The recovery time objective (RTO) for this operation mainly depends on the time it takes to deploy the and is typically below 30 min.

Cross AZ failure

See the section.

Region failure

Using Weka snapshots uploaded to S3 combined with S3 cross-region replication enables protection from an AWS region failure.

SSH keys rotation

For security reasons, it is advisable to rotate the SSH keys used for the EC2 instances.

To rotate the SSH keys, follow these steps:

• , and

• .

Related topic

In a WEKA cluster, the frontend container provides the default POSIX protocol, serving as the primary access point for the distributed filesystem. You can also define protocol containers for NFS, SMB, and S3 clients.

To configure protocol containers, you have two options for creating a cluster for the specified protocol:

1. Set up protocol services on existing backend servers.

2. Prepare additional dedicated servers for the protocol containers.

Dedicated filesystem requirement for cluster-wide persistent protocol configurations

It is required to have a dedicated filesystem that stores persistent protocol configurations. This filesystem is essential for coordinating coherent simultaneous access to files from multiple servers. It is advisable to assign a meaningful name to this configuration filesystem, such as .config_fs. Set the total capacity to 100 GB and avoid additional options like tiering and thin-provisioning.

Set up protocol containers on existing backend servers

With this option, you configure the existing cluster to provide the required protocol containers. The following topics guide you through the configuration for each protocol:

Prepare dedicated protocol servers

Using dedicated protocol servers enhances the cluster's capabilities and addresses diverse use cases. Each dedicated protocol server in the cluster can host one of these additional protocol containers alongside the existing frontend container.

These dedicated protocol servers function as complete and permanent members of the WEKA cluster. They run essential processes to access WEKA filesystems and incorporate switches supporting the protocols.

Dedicated protocol servers offer the following advantages:

• Optimized performance: Leverage dedicated CPU resources for tailored and efficient performance, optimizing overall resource usage.

• Independent protocol scaling: Scale specific protocols independently, mitigating resource contention and ensuring consistent performance across the cluster.

Procedure

1. Install the WEKA software on the dedicated protocol servers: Do one of the following:

   • Follow the default method as specified in .

   • Use the WEKA agent to install from a working backend. The following commands demonstrate this method:

1. Check the dedicated protocol servers: The servers join the cluster and can be verified using the command:

With dedicated protocol servers in place, the next step is to manage individual protocols.

Related topics

Overview

In tiered filesystems, the WEKA system optimizes storage efficiency and manages storage resources effectively by:

• Tiering only infrequently accessed portions of files (warm data), keeping hot data on SSDs.

• Efficiently bundling subsets of different files (to 64 MB objects) and tiering them to object stores, resulting in significant performance enhancements.

• Retrieving only the necessary data from the object store when accessing it, regardless of the entire object it was originally tiered with.

• Reclaiming logically freed data occurs when data is modified or deleted and is not used by any snapshots. Reclamation is a process of freeing up storage space that was previously allocated to data that is no longer needed.

SSD space reclamation in tiered filesystems

For logically freed data that resides on the SSD, the WEKA system immediately deletes the data from the SSD, leaving the physical space reclamation for the SSD erasure technique.

Object store space reclamation in tiered filesystems

Object store space reclamation is an important process that efficiently manages data stored on object storage.

WEKA organizes files into 64 MB objects for tiering. Each object can contain data from multiple files. Files smaller than 1 MB are consolidated into a single 64 MB object. For larger files, their parts are distributed across multiple objects. As a result, when a file is deleted (or updated and is not used by any snapshots), the space within one or more objects is marked as available for reclamation. However, the deletion of these objects only occurs under specific conditions.

Deleting related objects happens when all associated files are deleted, allowing for complete space reclamation within the object or during the reclamation process. Reclamation entails reading an eligible object from object storage and packing the active portions (representing data from undeleted files) with sections from other files that must be written to the object store. The resulting object is then written back to the object store, freeing up reclaimed space.

WEKA automates the reclamation process by monitoring the filesystems. When the reclaimable space within a filesystem exceeds 13%, the reclamation process begins. It continues until the total reclaimable space drops below 7%. This mechanism prevents write amplifications, allows time for higher portions of eligible 64 MB objects to become logically free, and prevents unnecessary object storage workload for small space reclamation. It's important to note that reclamation is only executed for objects with reclaimable space exceeding 5% within that object.

For regular filesystems where files are frequently deleted or updated, this behavior can result in the consumption of 7% to 13% more object store space than initially expected based on the total size of all files written to that filesystem. When planning object storage capacity or configuring usage alerts, it's essential to account for this additional space. Remember that this percentage may increase during periods of high object store usage or when data/snapshots are frequently deleted. Over time, it will return to the normal threshold as the load/burst is reduced.

View object store bucket capacity details

Run the weka fs tier capacity command to retrieve a comprehensive listing of data capacities associated with object store buckets per filesystem.

Example:

To list the data capacities of a specific filesystem, add the option --filesystem <filesystem name>.

Example:

Object tagging

When WEKA uploads objects to the object store, it assigns tags to categorize them. These tags are crucial because they enable the customer to implement specific lifecycle management rules in the object store based on the assigned tags.

For example, you can transfer objects of a specific filesystem when interacting with .

To enable upload tags, set it when adding or updating the object store bucket. For details, see the following:

• Using the GUI:

  • , or

  • by selecting Enable Upload Tags in the Advanced section.

• Using the CLI:

The following table indicates the additional tags WEKA adds to the object when using object tagging:

The object store must support S3 object-tagging and might require additional permissions to use object tagging.

For example, the following extra permissions are required in AWS S3:

• s3:PutObjectTagging

• s3:DeleteObjectTagging

Using the GUI, you can:

• Upload a snapshot

• Create a filesystem from an uploaded snapshot

• Sync a filesystem from a snapshot

Related topics

To learn about how to view, create, update, delete, and restore snapshots, see .

Upload a snapshot

You can upload a snapshot to a local, remote, or both object store buckets.

Procedure

1. From the menu, select Manage > Snapshots.

2. Select the three dots on the right of the required snapshot. From the menu, select Upload To Object Store.

1. A relevant message appears if a local or remote object store bucket is not attached to the filesystem. It enables opening a dialog to select an object store bucket and attach it to the filesystem. To add an object store, select Yes.

2. In the Attach Object Store to Filesystem dialog, select the object store bucket to attach the snapshot.

1. Select Save. The snapshot is uploaded to the target object store bucket.

2. Copy the snapshot locator:

   • Select the three dots on the right of the required snapshot, and select Copy Locator to Clipboard.

Related topics

Create a filesystem from an uploaded snapshot

You can create (or recreate) a filesystem from an uploaded snapshot, for example, when you need to migrate the filesystem data from one cluster to another.

When recreating a filesystem from a snapshot, adhere to the following guidelines:

• Pay attention to upload and download costs: Due to the bandwidth characteristics and potential costs when interacting with remote object stores, it is not allowed to download a filesystem from a remote object store bucket. If a snapshot on a local object store bucket exists, it is advisable to use that one. Otherwise, follow the procedure in the topic using the CLI.

• Use the same KMS master key: For an encrypted filesystem, to decrypt the snapshot data, use the same KMS master key as used in the encrypted filesystem. See the topic.

Before you begin

• Verify that the locator of the required snapshot (from the source cluster) is available (see the last step in the procedure).

• Ensure the object store is attached to the destination cluster.

Procedure

1. Connect to the destination cluster where you want to create the filesystem.

2. From the menu, select Manage > Filesystems, and select +Create.

3. In the Create Filesystem, do the following:

Related topics

Sync a filesystem from a snapshot

Media options for data storage in the WEKA system

In the WEKA system, data can be stored on two forms of media:

• On locally-attached SSDs, which are an integral part of the WEKA system configuration.

Pre-fetching API for data lifecycle management

Fetch files from an object store