> For the complete documentation index, see [llms.txt](https://docs.weka.io/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.weka.io/operation-guide/alerts/list-of-alerts-and-corrective-actions.md).

# List of alerts and corrective actions

Explore the list of system alerts, their meanings, and the recommended corrective actions. Understanding these alerts helps you respond to system events effectively.

For detailed maintenance and troubleshooting articles, search the WEKA Knowledge Base in the [WEKA support portal](https://support.weka.io/s/). Access to the portal requires a login and is available for registered customers and internal users. You can also contact the [Customer Success Team](/support/getting-support-for-your-weka-system.md#contact-customer-success-team).

| Alert Name                                  | Description                                                                                   | Corrective Action                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  | Severity |
| ------------------------------------------- | --------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------- |
| AdminDefaultPassword                        | Default admin password in use                                                                 | Change the admin password to restrict access to authorized users.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  | WARNING  |
| AgentNotRunning                             | Local agent is not running                                                                    | Restart the local agent on the specified server using the `weka agent restart` command.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            | DEBUG    |
| ApproachingClientsUnavailability            | Approaching connected clients limit                                                           | Ensure all backend containers are up. Alternatively, expand the cluster with additional backend containers or servers.                                                                                                                                                                                                                                                                                                                                                                                                                                                                             | DEBUG    |
| ApproachingSystemLimit                      | Approaching a system limit                                                                    | Follow the in-screen actions.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      | MAJOR    |
| AsymmetricPeerMTU                           | A process has peers with asymmetric MTU                                                       | Verify that switch ports, bonded interfaces, and NIC MTU settings are consistent across the path between this process and its peers.                                                                                                                                                                                                                                                                                                                                                                                                                                                               | WARNING  |
| AutoRemoveTimeoutTooLow                     | Stateless client auto-remove timeout too low                                                  | Remount the host with a higher auto-remove timeout value.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          | WARNING  |
| AvailableMemory                             | Not enough available memory                                                                   | Check the system memory usage.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     | MAJOR    |
| BackendNumaBalancingEnabled                 | NUMA balancing is enabled on a backend server                                                 | Disable automatic NUMA balancing on the backend server: `echo 0 > /proc/sys/kernel/numa_balancing`.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                | WARNING  |
| BackendVersionsMismatch                     | Backends mismatch cluster version                                                             | Upgrade all backends to match the cluster version.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 | WARNING  |
| BadDisksCapacityRatio                       | Large discrepancy between the smallest and largest drive                                      | Replace drives with comparable capacities to minimize the differences between the drives' sizes.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   | MAJOR    |
| BlockedJrpcMethod                           | JRPC method is blocked                                                                        | Unblock the JRPC method using the `blocked_jrpc_methods_remove` or `blocked_jrpc_methods_clear` manhole.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           | DEBUG    |
| BondInterfaceCompromised                    | Network high availability interface compromised                                               | Verify the network configuration, cables, and NICs.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                | MINOR    |
| BondMTUMismatch                             | Bond has inconsistent MTU across slaves                                                       | Inspect the host's interfaces with `ip link show` and align the MTU of all bond slaves with the bond master.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       | WARNING  |
| BucketCapacityExhausting                    | Buckets are nearing exhaustion of their maximum capacity                                      | Migrate to a cluster with a higher number of buckets.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              | DEBUG    |
| BucketHasNoQuorum                           | Too many compute processes are down                                                           | Ensure the compute processes on the containers {containers} are running and connected. If the issue persists, contact the Customer Success Team.                                                                                                                                                                                                                                                                                                                                                                                                                                                   | DEBUG    |
| BucketUnresponsive                          | Compute resource failure                                                                      | Check connectivity and drive status on the leader container. Ensure compute processes are running. If the issue persists, contact the Customer Success Team.                                                                                                                                                                                                                                                                                                                                                                                                                                       | CRITICAL |
| BucketsImbalance                            | FS buckets are not properly balanced between processes                                        | If this alert persists for a long time, contact the Customer Success Team for assistance.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          | WARNING  |
| CPUFrequentStarvation                       | CPU frequent starvation detected in the last minute                                           | Check the logs of the relevant containers for potential hardware or core allocation problems.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      | DEBUG    |
| CPUStarvation                               | CPU starvation detected in the last minute                                                    | Check the logs of the relevant containers for potential hardware problems. For a specific hang address, run `/weka/weka_addr2line` within the reported Weka container on the address to convert it into a symbol name.                                                                                                                                                                                                                                                                                                                                                                             | DEBUG    |
| CWTaskAbortionStuck                         | CWTask stuck in aborting state                                                                | Start IO to allow the task to complete the abortion process.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       | DEBUG    |
| CatalogIndexFsCapacityCritical              | Catalog index filesystem is critically low on space                                           | Increase the capacity of the catalog index filesystem to resume catalog ingestion.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 | CRITICAL |
| CatalogServiceDegraded                      | Catalog service is degraded                                                                   | Check the status of catalog service containers and ensure they are online with healthy frontend processes.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         | MAJOR    |
| CatalogServiceInactive                      | Catalog service is inactive                                                                   | Restore catalog service containers to bring the service back to active state.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      | CRITICAL |
| ChokingDetected                             | High congestion level                                                                         | Expand cluster resources if the workload maximizes current resources. For assistance, contact the Customer Success Team.                                                                                                                                                                                                                                                                                                                                                                                                                                                                           | DEBUG    |
| ClientVersionsMismatch                      | Clients mismatch cluster version                                                              | Upgrade clients to match the cluster version: `weka local upgrade`.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                | DEBUG    |
| ClockSkew                                   | Clock skew on server                                                                          | Ensure NTP is configured correctly on the containers and that their clocks are synchronized.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       | MINOR    |
| CloudHealth                                 | WEKA Home disconnected                                                                        | Check that the server has internet connectivity and is connected to WEKA Home. See the "WEKA Home - The WEKA support cloud" topic.                                                                                                                                                                                                                                                                                                                                                                                                                                                                 | MINOR    |
| CloudStatsError                             | Statistics upload failed                                                                      | See the event details in the System Events.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        | DEBUG    |
| ClusterInitializationError                  | Cluster initialization error                                                                  | Search for the underlying problem causing the error and act accordingly to start IO operations. To clear this alert, run the `weka cluster stop-io` command.                                                                                                                                                                                                                                                                                                                                                                                                                                       | WARNING  |
| ClusterIsUpgrading                          | Cluster upgrading                                                                             | Wait for the upgrade to complete. If the upgrade fails, contact the Customer Success Team.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         | DEBUG    |
| CompressedSpaceFragmented                   | Compressed free space is too fragmented                                                       | Check whether DATA\_DEFRAG tasks are paused or hanging. If this persists, contact the Customer Success Team.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       | WARNING  |
| ConfigOverridesActive                       | Configuration overrides are active                                                            | Contact the Customer Success Team.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 | DEBUG    |
| CoreOverlapping                             | Core overlapping                                                                              | Contact the Customer Success Team.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 | MAJOR    |
| DataIntegrity                               | Data integrity problem found                                                                  | The filesystem scan has identified data integrity problems that affect data accessibility or correctness. Contact the Customer Success Team.                                                                                                                                                                                                                                                                                                                                                                                                                                                       | CRITICAL |
| DataProtection                              | Partial data protection                                                                       | Restore full data protection immediately. If the cluster remains resilient to one failure, identify the down process, container, or drive and resolve the issue.                                                                                                                                                                                                                                                                                                                                                                                                                                   | MINOR    |
| DataServiceConfigurationNotSet              | Dataservice global configuration filesystem is not set                                        | Set the `config_fs` for dataservice by running the `weka dataservice global-config set --config-fs` command.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       | MINOR    |
| DedicatedWatchdog                           | Hardware watchdog driver required for dedicated server                                        | Install the driver by following the Knowledge Base instructions in the WEKA Support portal.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        | DEBUG    |
| DrainingStuck                               | Host stuck in draining state                                                                  | Check the host status and logs to identify blockers such as slow data migration, capacity constraints, or I/O errors. Verify cluster health and available capacity, then contact the Customer Success Team if the issue persists.                                                                                                                                                                                                                                                                                                                                                                  | MINOR    |
| DriveAutoEjected                            | Drive automatically ejected                                                                   | The drive was automatically ejected due to persistent issues. Monitor the drive phase-out progress and consider planning a replacement.                                                                                                                                                                                                                                                                                                                                                                                                                                                            | WARNING  |
| DriveAutoReactivating                       | Drive auto-reactivation                                                                       | Monitor the drive status.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          | WARNING  |
| DriveCriticalWarnings                       | Drive critical warnings                                                                       | Deactivate the drive using the `weka cluster drive deactivate` command and replace it.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             | MAJOR    |
| DriveDown                                   | Drive failed                                                                                  | Contact the Customer Success Team to assess drive health and determine if physical replacement is required. Provide the drive UUID, error message, and host details for faster diagnosis.                                                                                                                                                                                                                                                                                                                                                                                                          | MINOR    |
| DriveEndurancePercentageUsed                | Drive exceeds its life expectancy                                                             | Replace the specified drive before it fails.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       | MAJOR    |
| DriveEnduranceSparesRemaining               | Drive internal spares run too low                                                             | Replace the specified drive before it fails.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       | MAJOR    |
| DriveHasPartialFailureReports               | Some compute processes are failing to access disk                                             | Monitor the drive closely. If the issue persists, consider deactivating it using `weka cluster drive deactivate`.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  | MAJOR    |
| DriveNVKVRangesInsufficient                 | Drive has insufficient NVKV ranges                                                            | Contact the Customer Success Team.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 | MAJOR    |
| DriveNVKVRunningLow                         | Drive nearing exhaustion of internal resource                                                 | Contact the Customer Success Team.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 | DEBUG    |
| DriveNeedsPhaseout                          | A drive has too many errors                                                                   | Deactivate the drive using the `weka cluster drive deactivate {deactivate_flag}` command, and consider replacing it.                                                                                                                                                                                                                                                                                                                                                                                                                                                                               | MAJOR    |
| DriveUnwritableTooLong                      | Drive has been unwritable for too long                                                        | <p>A drive that has been unwritable for a long time risks NVKV overload when it becomes writable again.<br>Workaround: fully deactivate the drive, wait for it to reach the INACTIVE state, mark it writable, then activate it again.<br>If the issue persists, contact the Customer Success Team.</p>                                                                                                                                                                                                                                                                                             | MAJOR    |
| ExampleAlert                                | Example alert                                                                                 | Disable this alert by running the `set_example_alert_off` manhole.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 | DEBUG    |
| ExceptionsDuringAlertsEvaluation            | Exceptions thrown during alerts evaluation                                                    | Check the Assertion failures event, which may reveal the source of the problem. Contact the Customer Success Team if help is required.                                                                                                                                                                                                                                                                                                                                                                                                                                                             | DEBUG    |
| FaultsEnabled                               | Faults are enabled                                                                            | Contact the Customer Success Team.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 | DEBUG    |
| FilesystemKMSError                          | Filesystem KMS error                                                                          | Review the filesystem's KMS customization and the KMS configuration and connectivity.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              | DEBUG    |
| FilesystemsThinProvisioningLowSpace         | Filesystems thin provisioning low space                                                       | Consider adding more SSD capacity to the tenant containing these filesystems.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      | WARNING  |
| FilesystemsThinProvisioningReserveReached   | Filesystems thin provisioning capacity reserve reached                                        | Create a new filesystem or expand the filesystem's capacity using the reserved capacity.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           | DEBUG    |
| FsMetadataInconsistency                     | Filesystem metadata inconsistency found                                                       | The filesystem scan identified metadata inconsistencies, including anomalies in block metadata, snapshot structures, or upgrade-related data. These do not immediately affect data reads, but if left unresolved, they may impact filesystem reliability. Contact the Customer Success Team.                                                                                                                                                                                                                                                                                                       | MAJOR    |
| FsScanIssue                                 | Filesystem scan internal error detected                                                       | The filesystem scan encountered internal errors during validation, indicating a problem with the scan process rather than with the underlying data. Rerun the scan. If errors persist, contact the Customer Success Team.                                                                                                                                                                                                                                                                                                                                                                          | WARNING  |
| FsScanReclaimableSpace                      | Reclaimable storage space detected                                                            | The filesystem scan identified orphaned blocks or leaked storage space. These issues do not affect data integrity or accessibility, but represent space that can be reclaimed. They are typically caused by interrupted operations. Schedule a maintenance window to reclaim the affected space. If you need guidance, contact the Customer Success Team.                                                                                                                                                                                                                                          | MINOR    |
| GroupQuotasHardLimitReached                 | Group quota hard limit exceeded                                                               | Run the `weka fs quota list --group` cluster command to get the list of groups exceeding their hard quota limits. Clear some space for these groups or increase their hard quota limit.                                                                                                                                                                                                                                                                                                                                                                                                            | WARNING  |
| GroupQuotasSoftLimitReached                 | Group quota soft limit exceeded                                                               | Run the `weka fs quota list --group` cluster command to get the list of groups exceeding their soft quota limits. Clear some space for these groups or increase their soft quota limit.                                                                                                                                                                                                                                                                                                                                                                                                            | WARNING  |
| HAInterfaceMTUMismatch                      | HA group has inconsistent MTU across interfaces                                               | Inspect the host's DPDK interfaces and align the MTU of all interfaces of the same link layer.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     | WARNING  |
| HangingCacheSync                            | Cache sync is hanging                                                                         | Consider using `weka debug fs drop-dirty-cache` to drop the cache and enable other clients to access the file (unsynchronized writes will be lost).                                                                                                                                                                                                                                                                                                                                                                                                                                                | MINOR    |
| HangingClusterTasks                         | Cluster background task progress is hanging                                                   | To address a stalled task, contact the Customer Success Team.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      | DEBUG    |
| HangingIos                                  | Some IOs stop responding                                                                      | Ensure the compute processes are up, running, and connected. If a backend object store is configured, ensure it is connected and responsive. If the issue persists, contact the Customer Success Team.                                                                                                                                                                                                                                                                                                                                                                                             | DEBUG    |
| HardwareComponentFaulty                     | Hardware component fault detected                                                             | Check the management controller (BMC or IPMI) for diagnostic details and take corrective action.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   | MAJOR    |
| HardwareComponentWarning                    | Hardware component warning                                                                    | Check the management controller (BMC or IPMI) for details on the affected components.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              | WARNING  |
| HardwareMonitoringUnavailable               | Hardware monitor unavailable                                                                  | Verify that BMC or IPMI settings are configured correctly in the container resources. If the issue persists, contact Customer Success.                                                                                                                                                                                                                                                                                                                                                                                                                                                             | MINOR    |
| HighDrivesCapacity                          | SSD capacity overflow                                                                         | Free up space on the SSDs or add more SSDs to the cluster. See the "Expand specific resources of a container" topic.                                                                                                                                                                                                                                                                                                                                                                                                                                                                               | MAJOR    |
| HighLevelOfUnreclaimedCapacityInObjectStore | High level of unreclaimed space in an object store                                            | Check the object store connectivity and the progress of deletion operations. Validate the authorization of the deletion operations on the object store. Run the `weka fs tier capacity --filesystem {filesystem}` command for details.                                                                                                                                                                                                                                                                                                                                                             | DEBUG    |
| HighSSDToRAMRatio                           | High SSD to RAM ratio                                                                         | Consider increasing RAM cluster-wide, or removing unneeded drives, to meet filesystem (RAID) requirements and lower the SSD to RAM ratio.                                                                                                                                                                                                                                                                                                                                                                                                                                                          | WARNING  |
| HotspotInodes                               | IOs for some files have a long waiting queue                                                  | Retrieve the file path using `weka debug fs resolve-inode {inodeIdValue} --snap-view-id {snapViewIdValue}`. Get the list of clients with open handles using `weka debug fs lsof {inodeIdValue} --snap-view-id {snapViewIdValue}`. Understanding the access pattern may help you optimize the application's caching or modification mechanism.                                                                                                                                                                                                                                                      | DEBUG    |
| IBNotEnhanced                               | Enhanced IB mode disabled                                                                     | Contact the Customer Success Team to correct this issue.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           | DEBUG    |
| ImbalancedCpuUsage                          | Imbalanced CPU usage detected in cluster processes                                            | Examine the system configuration for abnormalities that may be causing the CPU usage imbalance.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    | DEBUG    |
| JumboConnectivity                           | A container cannot send jumbo frames                                                          | Check the container's network settings and the switch it is connected to, and ensure jumbo frames are enabled. This setting improves performance.                                                                                                                                                                                                                                                                                                                                                                                                                                                  | WARNING  |
| KMSError                                    | KMS error                                                                                     | Review the KMS configuration and connectivity.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     | MAJOR    |
| LeaderPreparedForUpgrade                    | Leader prepared for upgrade                                                                   | After the upgrade, the leader state automatically returns to normal. If the issue persists, contact the Customer Success Team.                                                                                                                                                                                                                                                                                                                                                                                                                                                                     | DEBUG    |
| LegacyManualOverridesActive                 | Legacy manual overrides are active                                                            | Contact the Customer Success Team.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 | DEBUG    |
| LicenseError                                | License error                                                                                 | <p>Ensure the cluster uses the correct license, that the license has not expired, and that the allocated space does not exceed the license limits.<br>The <code>LicenseError</code> alert includes a 5% grace threshold. It is triggered only when used capacity exceeds 105% of the licensed limit, which allows for minor overages without generating an immediate alert.</p>                                                                                                                                                                                                                    | WARNING  |
| LocalTLSCertificateExpired                  | Local TLS certificate expired                                                                 | Update the local certificate.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      | DEBUG    |
| LocalTLSCertificateExpiringSoon             | Local TLS certificate is expiring soon                                                        | Update the local certificate.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      | DEBUG    |
| LocalTLSConnectivityToNeighbors             | Outgoing TLS connectivity to backends is down                                                 | Fix the TLS issue. One possibility is an error in the local cacert in `/etc/wekaio/certs`.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         | DEBUG    |
| LongestWaitInodes                           | Elevated IO wait time detected for some files                                                 | Retrieve the file path using `weka debug fs resolve-inode {inodeIdValue} --snap-view-id {snapViewIdValue}`. Get the list of clients with open handles using `weka debug fs lsof {inodeIdValue} --snap-view-id {snapViewIdValue}`. Understanding the access pattern may help you optimize the application's caching or modification mechanism.                                                                                                                                                                                                                                                      | DEBUG    |
| LowDiskSpace                                | Low disk space                                                                                | See the event details in the System Events.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        | MINOR    |
| ManualOverridesActive                       | Manual overrides are active                                                                   | Contact the Customer Success Team.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 | DEBUG    |
| ManualOverridesForced                       | Manual overrides are forced                                                                   | Contact the Customer Success Team.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 | DEBUG    |
| MismatchedDriveFailureDomain                | A drive's failure domain does not match the failure domain of the container it is attached to | Attach the drive to a container with a matching failure domain, or re-provision the drive to clear its failure-domain assignment.                                                                                                                                                                                                                                                                                                                                                                                                                                                                  | MAJOR    |
| MismatchedJoinSecrets                       | Backend containers do not have the same join secrets                                          | This may create problems rejoining or reforming the cluster. Ensure all backend containers have the same join secrets.                                                                                                                                                                                                                                                                                                                                                                                                                                                                             | DEBUG    |
| NegativeUnprovisionedCapacity               | Negative unprovisioned capacity                                                               | Resize one or more of the filesystems to reclaim capacity. For more information, contact the Customer Success Team.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                | DEBUG    |
| NetworkFailedToStartPorts                   | Network ports failed to start                                                                 | Run the `weka debug net ports $NODE` command to see the current status.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            | DEBUG    |
| NetworkInterfaceLinkDown                    | Network interface link status down                                                            | Check the connectivity to the specified network interface and verify that nothing is blocking it.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  | MINOR    |
| NfsLocksDisabled                            | NFS locks disabled                                                                            | Configure `config_fs` using the `weka nfs global-config set --config-fs=<filesystem name>` command.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                | WARNING  |
| NfsServiceDownAlert                         | NFS service down                                                                              | If down services persist, contact the Customer Success Team.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       | MAJOR    |
| NicPciSettingsMisconfigured                 | NIC PCI firmware settings suboptimal                                                          | <p>The NIC firmware setting(s) {settings} are not set to 1. These settings improve PCIe performance for high-throughput workloads. Applying them requires resetting the NIC firmware. ADVANCED\_PCI\_SETTINGS is card-wide; PCI\_WR\_ORDERING is per-port, so this command must be run for each PCI address the alert lists.<br>On the affected host, stop the Weka containers, then run:<br><code>weka local run mstconfig -d {pciAddr} set ADVANCED\_PCI\_SETTINGS=1 PCI\_WR\_ORDERING=1</code><br><code>weka local run mstfwreset -d {pciAddr} reset</code><br>Restart the Weka containers.</p> | WARNING  |
| NoCgroupsConfigured                         | No cgroups configured warnings                                                                | Enable and configure cgroups (v1/v2) by following the Cgroups configuration section in the documentation.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          | WARNING  |
| NoClusterLicense                            | No license assigned                                                                           | Obtain and install a license from `get.weka.io`.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   | WARNING  |
| NoHotSpareFailureDomains                    | No hot-spare failure domains have been configured                                             | Assign one or more hot-spare failure domains by running the `weka cluster hot-spare` command.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      | WARNING  |
| NodeBlacklisted                             | A process cannot rejoin the cluster                                                           | To enable the process to rejoin the cluster, allow it by running the `weka debug blacklist remove` command.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        | DEBUG    |
| NodeDisconnected                            | Process disconnected                                                                          | Check the network connectivity to ensure the processes can communicate with the cluster.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           | MINOR    |
| NodeNetworkUnstable                         | A process with an unstable network detected                                                   | Ensure proper network connectivity in the cluster. If the problem is not resolved, contact the Customer Success Team.                                                                                                                                                                                                                                                                                                                                                                                                                                                                              | WARNING  |
| NodeRDMANotActive                           | RDMA support for process is inactive                                                          | Ensure that at least one RDMA-capable device exists.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               | DEBUG    |
| NodeTieringConnectivity                     | A process cannot connect to an object store                                                   | A process cannot connect to the object store due to network connectivity issues, process health issues, or the object storage vendor equipment itself. Check the connectivity with the object store and ensure the process can communicate with it.                                                                                                                                                                                                                                                                                                                                                | MAJOR    |
| NonTlsApisAllowed                           | Non-TLS APIs are allowed                                                                      | Update the TLS strictness setting to enforce encrypted TLS APIs over HTTP.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         | DEBUG    |
| NotEnoughActiveDrives                       | Reduced data protection                                                                       | Check the connectivity and server status, and activate drives in more failure domains.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             | MAJOR    |
| NotEnoughSSDCapacity                        | Some provisioned capacity is unavailable due to failed drives                                 | Check for down drives.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             | MAJOR    |
| NotificationQueueHighLoad                   | S3 notification queue reached high watermark                                                  | Reduce the notification load or increase the queue capacity. If the issue persists, contact the Customer Success Team.                                                                                                                                                                                                                                                                                                                                                                                                                                                                             | MINOR    |
| NotificationSendFailure                     | S3 notification send failure                                                                  | Check system logs for details and restore Kafka service availability.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              | MAJOR    |
| PartialConnectivityTrackingDisabled         | Partial connectivity tracking is disabled                                                     | To re-enable partial connectivity tracking, contact the Customer Success Team.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     | DEBUG    |
| PartialHugepageAllocation                   | Not enough memory                                                                             | Check the system's hugepage and memory configuration.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              | MAJOR    |
| PartiallyConnectedNode                      | A partially connected process detected                                                        | Ensure proper network connectivity in the cluster. If the issue persists, contact the Customer Success Team.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       | MINOR    |
| PassedClientsAvailabilityThreshold          | Reached connected clients limit                                                               | Add more backend containers or servers to the cluster, check whether the backends are down, or disconnect some clients.                                                                                                                                                                                                                                                                                                                                                                                                                                                                            | DEBUG    |
| PathsDegraded                               | Degraded paths                                                                                | Contact the Customer Success Team to review path connectivity.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     | MINOR    |
| PerformanceDegradedLowRAM                   | Low server RAM                                                                                | Add more servers to the cluster, add RAM to the backend servers, or increase the memory allocation to the compute processes.                                                                                                                                                                                                                                                                                                                                                                                                                                                                       | MAJOR    |
| QuotasHardLimitReached                      | Directory quota hard limit exceeded                                                           | Run the `weka fs quota list` cluster command to get the list of directories exceeding their hard quota limits. Clear some space for these directories or increase their hard quota limit.                                                                                                                                                                                                                                                                                                                                                                                                          | WARNING  |
| QuotasSoftLimitReached                      | Directory quota soft limit exceeded                                                           | Run the `weka fs quota list` cluster command to get the list of directories exceeding their soft quota limits. Clear some space for these directories or increase their soft quota limit.                                                                                                                                                                                                                                                                                                                                                                                                          | WARNING  |
| RAIDCapacityExhaustion                      | RAID capacity nearing exhaustion                                                              | Monitor the situation for approximately 5 minutes to allow for automatic rebalancing. If RAID capacity usage remains high or continues to increase, contact the Customer Success Team immediately to prevent write failures.                                                                                                                                                                                                                                                                                                                                                                       | MAJOR    |
| RequestedActionFailure                      | Requested action failed                                                                       | Check system logs for details and verify host accessibility, resource availability, and system state before retrying.                                                                                                                                                                                                                                                                                                                                                                                                                                                                              | DEBUG    |
| RequestedActionTimeout                      | Container has requested action timeout                                                        | Check the container status and logs for more information about the timeout.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        | DEBUG    |
| ResourcesNotApplied                         | Resource changes not applied                                                                  | Apply the resource changes by running the `weka cluster container apply <host-id>` command.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        | DEBUG    |
| S3LifecycleTaskCannotStart                  | S3 lifecycle tasks cannot start - missing dataservice containers                              | Set up dataservice containers by following the "Set up a Data Services container for background tasks" documentation.                                                                                                                                                                                                                                                                                                                                                                                                                                                                              | MINOR    |
| S3UpgradeInProgress                         | S3 is undergoing a software upgrade                                                           | If the upgrade doesn't finish successfully, contact the Customer Success Team.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     | WARNING  |
| S3VersioningNoDataservIlm                   | Versioning configured but no active dataserv container with ILM configured                    | <p>Ensure dataserv containers are running, enable the S3 lifecycle task manager, and add a cleanup rule:<br><code>weka dataservice s3-lifecycle-task enable</code><br><code>weka s3 bucket lifecycle-rule add \<bucket> \<days> --noncurrent</code></p>                                                                                                                                                                                                                                                                                                                                            | MINOR    |
| S3VersioningNoNoncurrentExpirationRule      | Versioning configured but no NoncurrentVersionExpiration rule                                 | <p>Add a cleanup rule with:<br><code>weka s3 bucket lifecycle-rule add \<bucket> \<days> --noncurrent</code></p>                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   | MINOR    |
| SSDCapacityDiscrepancy                      | Mismatch between actual and expected SSD capacity usage                                       | To address this mismatch, monitor compute process stability and contact the Customer Success Team.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 | DEBUG    |
| SSDCapacityTooHigh                          | Available capacity cannot be fully utilized                                                   | To optimize the SSD capacity usage, contact the Customer Success Team.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             | WARNING  |
| SystemDefinedTLS                            | TLS certificate is not user-defined                                                           | Replace the auto-generated self-signed certificate with a user-defined certificate by running the `weka security tls set` command.                                                                                                                                                                                                                                                                                                                                                                                                                                                                 | WARNING  |
| SystemdSafeShutdownConfigMismatch           | Systemd safe shutdown configuration differs across cluster servers                            | Enable systemd safe shutdown on all servers using `weka local enable-safe-shutdown`.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               | WARNING  |
| TLSCertificateExpired                       | TLS certificate expired                                                                       | Replace the existing certificate using the `weka security tls set` command.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        | MAJOR    |
| TLSCertificateExpiresSoon                   | TLS certificate is about to expire                                                            | Replace the existing certificate using the `weka security tls set` command.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        | MAJOR    |
| TelemetryStatusFault                        | Telemetry status is not streaming                                                             | Check your telemetry sinks configuration.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          | WARNING  |
| TieredFilesystemOverfillingSSD              | Tiered filesystem SSD capacity threshold exceeded                                             | Verify tiering service connectivity to object storage and check for network or authentication issues blocking data migration. Consider increasing tiering upload bandwidth or reviewing tiering policies and thresholds if consistently near capacity.                                                                                                                                                                                                                                                                                                                                             | WARNING  |
| TooManyPendingClusterwideJobs               | Too many pending cluster-wide jobs                                                            | Consider changing the policy configuration.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        | DEBUG    |
| TraceDumperDown                             | Trace dumper is down                                                                          | To restart the trace dumper, contact the Customer Success Team.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    | DEBUG    |
| TracesDisabled                              | Traces are disabled                                                                           | To re-enable the cluster traces, run the `weka debug traces start` command.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        | DEBUG    |
| TracesFreezePeriodActive                    | Trace freeze is active                                                                        | If the issue persists after resolution, contact the Customer Success Team to deactivate the trace freeze.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          | DEBUG    |
| UdpModePerformanceWarning                   | Backend container configured in UDP mode                                                      | If this is a misconfiguration, add network devices to the specified backend container using the `weka cluster container net add` command.                                                                                                                                                                                                                                                                                                                                                                                                                                                          | DEBUG    |
| UnstableHosts                               | Host marked unstable during upgrade                                                           | Review container status and logs to identify and resolve hardware, network, or service problems before retrying the upgrade.                                                                                                                                                                                                                                                                                                                                                                                                                                                                       | DEBUG    |
| UnwritableDisksConfigured                   | A drive is set to unwritable                                                                  | If the drive remains unwritable after maintenance, contact the Customer Success Team.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              | DEBUG    |
| UserQuotasHardLimitReached                  | User quota hard limit exceeded                                                                | Run the `weka fs quota list --user` cluster command to get the list of users exceeding their hard quota limits. Clear some space for these users or increase their hard quota limit.                                                                                                                                                                                                                                                                                                                                                                                                               | WARNING  |
| UserQuotasSoftLimitReached                  | User quota soft limit exceeded                                                                | Run the `weka fs quota list --user` cluster command to get the list of users exceeding their soft quota limits. Clear some space for these users or increase their soft quota limit.                                                                                                                                                                                                                                                                                                                                                                                                               | WARNING  |
| WTracerDaemonWriteIOFailures                | WTracer daemon write I/O failures detected                                                    | Review System Events for details and check for storage, network, or disk issues affecting trace data writes.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       | WARNING  |
| WTracerLostTraces                           | WTracer lost traces detected                                                                  | Review System Events and check for high system load, memory pressure, or I/O bottlenecks.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          | MINOR    |


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter:

```
GET https://docs.weka.io/operation-guide/alerts/list-of-alerts-and-corrective-actions.md?ask=<question>&goal=<endgoal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.
