W E K A
3.14
3.14
  • WEKA v3.14 Documentation
  • Weka System Overview
    • About the WEKA System
    • SSD Capacity Management
    • Filesystems, Object Stores & Filesystem Groups
    • Weka Networking
    • Data Lifecycle Management
    • Weka Client & Mount Modes
    • Glossary
  • Getting Started with Weka
    • Quick Install Guide
    • Managing the Weka System
    • CLI Overview
    • GUI Overview
    • Serving IOs with WekaFS
  • Planning & Installation
    • Prerequisites for Installation
    • Bare Metal Installation
      • Planning a Weka System Installation
      • Setting Up the Hosts
        • SR-IOV Enablement
      • Obtaining the Weka Install File
      • Weka System Installation Process Using the CLI
      • Adding Clients
    • AWS Installation
      • Self-Service Portal
      • CloudFormation Template Generator
      • Deployment Types
      • AWS Outposts Deployment
      • Supported EC2 Instance Types
      • Adding Clients
      • Auto Scaling Group
      • Troubleshooting
  • Performance
    • Testing Weka Performance
      • Test Environment Details
  • WekaFS Filesystems
    • Managing Filesystems, Object Stores & Filesystem Groups
      • Managing Object Stores
      • Managing Filesystem Groups
      • Managing Filesystems
      • Attaching/Detaching Object Stores to/from Filesystems
      • KMS Management
    • Advanced Data Lifecycle Management
      • Advanced Time-based Policies for Data Storage Location
      • Data Management in Tiered Filesystems
      • Transition Between Tiered and SSD-Only Filesystems
      • Manual fetch and release of data
    • Mounting Filesystems
    • Snapshots
    • Snap-To-Object
    • Quota Management
  • Additional Protocols
    • NFS
    • SMB
      • SMB Management Using CLIs
      • SMB Management Using the GUI
    • S3
      • S3 Cluster Management
      • S3 Buckets Management
      • S3 Users and Authentication
      • S3 Information Lifecycle Management
      • Audit S3 APIs
      • S3 Limitations
      • S3 Examples using boto3
  • Operation Guide
    • Alerts
      • List of Alerts
    • Events
      • List of Events
    • Statistics
      • List of Statistics
    • System Congestion
    • Security
      • User Management
      • Organizations
    • Expanding & Shrinking Cluster Resources
      • Expand & Shrink Overview
      • Stages in Adding a Backend Host
      • Expansion of Specific Resources
      • Shrinking a Cluster
    • Background Tasks
    • Upgrading Weka Versions
  • Billing & Licensing
    • License Overview
    • Classic License
    • Pay-As-You-Go License
  • Support
    • Prerequisites and Compatibility
    • Getting Support for Your Weka System
    • The Weka Support Cloud
    • Diagnostics CLI Command
  • Appendix
    • Weka CSI Plugin
    • External Monitoring
    • Snapshot Management
  • REST API
Powered by GitBook
On this page
  • Overview
  • System Congestion Events/Alerts
  1. Operation Guide

System Congestion

This page describes possible congestion issues in the Weka system.

PreviousList of StatisticsNextSecurity

Last updated 3 years ago

Overview

The Weka system is built to be very efficient, provide maximum performance and saturate the network links.

In some situations, the system itself may slow down IOs when reaching some limits (or even block new IOs at higher limits) until the congested resource is relieved. Such situations may be transient and the issue will be resolved on its own after a short time. However, there are also cases that suggest an issue that needs to be addressed, such as a workload maxing out the resources of the cluster. In such cases, the cluster resources must be expanded, as described in . Contact the Weka Support Team for more information on this.

System Congestion Events/Alerts

The Weka system can issue several types of congestion events/alerts:

Type

Description

Actions

FIBERS

Extreme load of concurrent system operations on a node

This is usually a transient situation due to load on them system. If the load is consistent and the problem persists, add more resources (hosts/cores), as described in or .

DESTAGER

Too many pending IOs are waiting to be written for a specific node

This is usually a transient situation due to load on them system. If the load is consistent and the problem persists, add more hosts to the cluster as described in , or expand the host resources as described in .

SSD

Too many pending IOs to the SSD

If there is a single SSD, it is probably faulty and needs to be replaced. If there are multiple SSDs, the load on the system is too high. To handle such a load, more SSDs should be added to the system, as described in .

RAID_NOT_OK

More IO failures than can be handled have occurred, and IOs cannot be served

Make sure to bring up any host that might be down. If all hosts are up, contact the Weka Support Team.

XDESTAGE

Auxiliary cluster resources are low

This is usually a transient situation due to load on the system. If the load is consistent and the problem persists, add more hosts to the cluster as described in , or consult the Weka Support Team.

Expanding & Shrinking Cluster Resources
Adding a Backend Host
Adding a Backend Host
Expansion of Specific Resources
Adding a Backend Host
Addition of CPU cores
Expansion of Specific Resources