> For the complete documentation index, see [llms.txt](https://docs.weka.io/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.weka.io/operation-guide/alerts/list-of-alerts-and-corrective-actions.md). # List of alerts and corrective actions Explore the list of system alerts, their meanings, and the recommended corrective actions. Understanding these alerts helps you respond to system events effectively. This list covers alerts of WARNING severity and above. Alerts with DEBUG severity are informational and are not shown by default; to see them on a cluster, run `weka alerts --severity DEBUG`. For detailed maintenance and troubleshooting articles, search the WEKA Knowledge Base in the [WEKA support portal](https://support.weka.io/s/). Access to the portal requires a login and is available for registered customers and internal users. You can also contact the [Customer Success Team](/support/getting-support-for-your-weka-system.md#contact-customer-success-team). | Alert Name | Description | Corrective Action | Severity | | -------------------------------------- | --------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------- | | AdminDefaultPassword | Default admin password in use | Change the admin password to restrict access to authorized users. | WARNING | | ApproachingSystemLimit | Approaching a system limit | Follow the in-screen actions. | MAJOR | | AsymmetricPeerMTU | A process has peers with asymmetric MTU | Verify that switch ports, bonded interfaces, and NIC MTU settings are consistent across the path between this process and its peers. | WARNING | | AutoRemoveTimeoutTooLow | Stateless client auto-remove timeout too low | Remount the host with a higher auto-remove timeout value. | WARNING | | AvailableMemory | Not enough available memory | Check the system memory usage. | MAJOR | | BackendNumaBalancingEnabled | NUMA balancing is enabled on a backend server | Disable automatic NUMA balancing on the backend server: `echo 0 > /proc/sys/kernel/numa_balancing`. | WARNING | | BackendVersionsMismatch | Backends mismatch cluster version | Upgrade all backends to match the cluster version. | WARNING | | BadDisksCapacityRatio | Large discrepancy between the smallest and largest drive | Replace drives with comparable capacities to minimize the differences between the drives' sizes. | MAJOR | | BondInterfaceCompromised | Network high availability interface compromised | Verify the network configuration, cables, and NICs. | MINOR | | BondMTUMismatch | Bond has inconsistent MTU across slaves | Inspect the host's interfaces with `ip link show` and align the MTU of all bond slaves with the bond master. | WARNING | | BucketUnresponsive | Compute resource failure | Check connectivity and drive status on the leader container. Ensure compute processes are running. If the issue persists, contact the Customer Success Team. | CRITICAL | | BucketsImbalance | FS buckets are not properly balanced between processes | If this alert persists for a long time, contact the Customer Success Team for assistance. | WARNING | | CatalogIndexFsCapacityCritical | Catalog index filesystem is critically low on space | Increase the capacity of the catalog index filesystem to resume catalog ingestion. | CRITICAL | | CatalogServiceDegraded | Catalog service is degraded | Check the status of catalog service containers and ensure they are online with healthy frontend processes. | MAJOR | | CatalogServiceInactive | Catalog service is inactive | Restore catalog service containers to bring the service back to active state. | CRITICAL | | ClockSkew | Clock skew on server | Ensure NTP is configured correctly on the containers and that their clocks are synchronized. | MINOR | | CloudHealth | WEKA Home disconnected | Check that the server has internet connectivity and is connected to WEKA Home. See the "WEKA Home - The WEKA support cloud" topic. | MINOR | | ClusterInitializationError | Cluster initialization error | Search for the underlying problem causing the error and act accordingly to start IO operations. To clear this alert, run the `weka cluster stop-io` command. | WARNING | | CompressedSpaceFragmented | Compressed free space is too fragmented | Check whether DATA\_DEFRAG tasks are paused or hanging. If this persists, contact the Customer Success Team. | WARNING | | CoreOverlapping | Core overlapping | Contact the Customer Success Team. | MAJOR | | DataIntegrity | Data integrity problem found | The filesystem scan has identified data integrity problems that affect data accessibility or correctness. Contact the Customer Success Team. | CRITICAL | | DataProtection | Partial data protection | Restore full data protection immediately. If the cluster remains resilient to one failure, identify the down process, container, or drive and resolve the issue. | MINOR | | DataServiceConfigurationNotSet | Dataservice global configuration filesystem is not set | Set the `config_fs` for dataservice by running the `weka dataservice global-config set --config-fs` command. | MINOR | | DrainingStuck | Host stuck in draining state | Check the host status and logs to identify blockers such as slow data migration, capacity constraints, or I/O errors. Verify cluster health and available capacity, then contact the Customer Success Team if the issue persists. | MINOR | | DriveAutoEjected | Drive automatically ejected | The drive was automatically ejected due to persistent issues. Monitor the drive phase-out progress and consider planning a replacement. | WARNING | | DriveAutoReactivating | Drive auto-reactivation | Monitor the drive status. | WARNING | | DriveCriticalWarnings | Drive critical warnings | Deactivate the drive using the `weka cluster drive deactivate` command and replace it. | MAJOR | | DriveDown | Drive failed | Contact the Customer Success Team to assess drive health and determine if physical replacement is required. Provide the drive UUID, error message, and host details for faster diagnosis. | MINOR | | DriveEndurancePercentageUsed | Drive exceeds its life expectancy | Replace the specified drive before it fails. | MAJOR | | DriveEnduranceSparesRemaining | Drive internal spares run too low | Replace the specified drive before it fails. | MAJOR | | DriveHasPartialFailureReports | Some compute processes are failing to access disk | Monitor the drive closely. If the issue persists, consider deactivating it using `weka cluster drive deactivate`. | MAJOR | | DriveNVKVRangesInsufficient | Drive has insufficient NVKV ranges | Contact the Customer Success Team. | MAJOR | | DriveNeedsPhaseout | A drive has too many errors | Deactivate the affected drive using the `weka cluster drive deactivate ` command; add `--hard` if automatic ejection failed. Consider replacing the drive. | MAJOR | | DriveUnwritableTooLong | Drive has been unwritable for too long |

A drive that has been unwritable for a long time risks NVKV overload when it becomes writable again.
Workaround: fully deactivate the drive, wait for it to reach the INACTIVE state, mark it writable, then activate it again.
If the issue persists, contact the Customer Success Team.

| MAJOR | | FilesystemsThinProvisioningLowSpace | Filesystems thin provisioning low space | Consider adding more SSD capacity to the tenant containing these filesystems. | WARNING | | FsMetadataInconsistency | Filesystem metadata inconsistency found | The filesystem scan identified metadata inconsistencies, including anomalies in block metadata, snapshot structures, or upgrade-related data. These do not immediately affect data reads, but if left unresolved, they may impact filesystem reliability. Contact the Customer Success Team. | MAJOR | | FsScanIssue | Filesystem scan internal error detected | The filesystem scan encountered internal errors during validation, indicating a problem with the scan process rather than with the underlying data. Rerun the scan. If errors persist, contact the Customer Success Team. | WARNING | | FsScanReclaimableSpace | Reclaimable storage space detected | The filesystem scan identified orphaned blocks or leaked storage space. These issues do not affect data integrity or accessibility, but represent space that can be reclaimed. They are typically caused by interrupted operations. Schedule a maintenance window to reclaim the affected space. If you need guidance, contact the Customer Success Team. | MINOR | | GroupQuotasHardLimitReached | Group quota hard limit exceeded | Run the `weka fs quota list --type group` cluster command to get the list of groups exceeding their hard quota limits. Clear some space for these groups or increase their hard quota limit. | WARNING | | GroupQuotasSoftLimitReached | Group quota soft limit exceeded | Run the `weka fs quota list --type group` cluster command to get the list of groups exceeding their soft quota limits. Clear some space for these groups or increase their soft quota limit. | WARNING | | HAInterfaceMTUMismatch | HA group has inconsistent MTU across interfaces | Inspect the host's DPDK interfaces and align the MTU of all interfaces of the same link layer. | WARNING | | HangingCacheSync | Cache sync is hanging | Consider using `weka debug fs drop-dirty-cache` to drop the cache and enable other clients to access the file (unsynchronized writes will be lost). | MINOR | | HardwareComponentFaulty | Hardware component fault detected | Check the management controller (BMC or IPMI) for diagnostic details and take corrective action. | MAJOR | | HardwareComponentWarning | Hardware component warning | Check the management controller (BMC or IPMI) for details on the affected components. | WARNING | | HardwareMonitoringUnavailable | Hardware monitor unavailable | Verify that BMC or IPMI settings are configured correctly in the container resources. If the issue persists, contact Customer Success. | MINOR | | HighDrivesCapacity | SSD capacity overflow | Free up space on the SSDs or add more SSDs to the cluster. See the "Expand specific resources of a container" topic. | MAJOR | | HighSSDToRAMRatio | High SSD to RAM ratio | Consider increasing RAM cluster-wide, or removing unneeded drives, to meet filesystem (RAID) requirements and lower the SSD to RAM ratio. | WARNING | | JumboConnectivity | A container cannot send jumbo frames | Check the container's network settings and the switch it is connected to, and ensure jumbo frames are enabled. This setting improves performance. | WARNING | | KMSError | KMS error | Review the KMS configuration and connectivity. | MAJOR | | LicenseError | License error |

Ensure the cluster uses the correct license, that the license has not expired, and that the allocated space does not exceed the license limits.
The LicenseError alert includes a 5% grace threshold. It is triggered only when used capacity exceeds 105% of the licensed limit, which allows for minor overages without generating an immediate alert.

| WARNING | | LowDiskSpace | Low disk space | See the event details in the System Events. | MINOR | | MismatchedDriveFailureDomain | A drive's failure domain does not match the failure domain of the container it is attached to | Attach the drive to a container with a matching failure domain, or re-provision the drive to clear its failure-domain assignment. | MAJOR | | MismatchedJoinSecrets | Backend containers do not have the same join secrets | This may create problems rejoining or reforming the cluster. Ensure all backend containers have the same join secrets. | WARNING | | NetworkInterfaceLinkDown | Network interface link status down | Check the connectivity to the specified network interface and verify that nothing is blocking it. | MINOR | | NetworkSpaceApplyStuck | Network Space apply stuck | Inspect with `weka debug netspace apply-status`. If a backend is permanently down, remove it from the cluster, or force the operation to terminal with `weka debug netspace force-complete`. | MAJOR | | NfsLocksDisabled | NFS locks disabled | Configure `config_fs` using the `weka nfs global-config set --config-fs=` command. | WARNING | | NfsServiceDownAlert | NFS service down | If down services persist, contact the Customer Success Team. | MAJOR | | NicPciSettingsMisconfigured | NIC PCI firmware settings suboptimal |

The NIC firmware setting(s) {settings} are not set to 1. These settings improve PCIe performance for high-throughput workloads. Applying them requires resetting the NIC firmware. ADVANCED\_PCI\_SETTINGS is card-wide; PCI\_WR\_ORDERING is per-port, so this command must be run for each PCI address the alert lists.
On the affected host, stop the Weka containers, then run:
weka local run mstconfig -d {pciAddr} set ADVANCED\_PCI\_SETTINGS=1 PCI\_WR\_ORDERING=1
weka local run mstfwreset -d {pciAddr} reset
Restart the Weka containers.

| WARNING | | NoCgroupsConfigured | No cgroups configured warnings | Enable and configure cgroups (v1/v2) by following the Cgroups configuration section in the documentation. | WARNING | | NoClusterLicense | No license assigned | Obtain and install a license from `get.weka.io`. | WARNING | | NoHotSpareFailureDomains | No hot-spare failure domains have been configured | Assign one or more hot-spare failure domains by running the `weka cluster hot-spare` command. | WARNING | | NodeDisconnected | Process disconnected | Check the network connectivity to ensure the processes can communicate with the cluster. | MINOR | | NodeNetworkUnstable | A process with an unstable network detected | Ensure proper network connectivity in the cluster. If the problem is not resolved, contact the Customer Success Team. | WARNING | | NodeTieringConnectivity | A process cannot connect to an object store | A process cannot connect to the object store due to network connectivity issues, process health issues, or the object storage vendor equipment itself. Check the connectivity with the object store and ensure the process can communicate with it. | MAJOR | | NotEnoughActiveDrives | Reduced data protection | Check the connectivity and server status, and activate drives in more failure domains. | MAJOR | | NotEnoughSSDCapacity | Some provisioned capacity is unavailable due to failed drives | Check for down drives. | MAJOR | | NotificationQueueHighLoad | S3 notification queue reached high watermark | Reduce the notification load or increase the queue capacity. If the issue persists, contact the Customer Success Team. | MINOR | | NotificationSendFailure | S3 notification send failure | Check system logs for details and restore Kafka service availability. | MAJOR | | PartialHugepageAllocation | Not enough memory | Check the system's hugepage and memory configuration. | MAJOR | | PartiallyConnectedNode | A partially connected process detected | Ensure proper network connectivity in the cluster. If the issue persists, contact the Customer Success Team. | MINOR | | PathsDegraded | Degraded paths | Contact the Customer Success Team to review path connectivity. | MINOR | | PerformanceDegradedLowRAM | Low server RAM | Add more servers to the cluster, add RAM to the backend servers, or increase the memory allocation to the compute processes. | MAJOR | | QuotasHardLimitReached | Directory quota hard limit exceeded | Run the `weka fs quota list` cluster command to get the list of directories exceeding their hard quota limits. Clear some space for these directories or increase their hard quota limit. | WARNING | | QuotasSoftLimitReached | Directory quota soft limit exceeded | Run the `weka fs quota list` cluster command to get the list of directories exceeding their soft quota limits. Clear some space for these directories or increase their soft quota limit. | WARNING | | RAIDCapacityExhaustion | RAID capacity nearing exhaustion | Monitor the situation for approximately 5 minutes to allow for automatic rebalancing. If RAID capacity usage remains high or continues to increase, contact the Customer Success Team immediately to prevent write failures. | MAJOR | | S3LifecycleTaskCannotStart | S3 lifecycle tasks cannot start - missing dataservice containers | Set up dataservice containers by following the "Set up a Data Services container for background tasks" documentation. | MINOR | | S3UpgradeInProgress | S3 is undergoing a software upgrade | If the upgrade doesn't finish successfully, contact the Customer Success Team. | WARNING | | S3VersioningNoDataservIlm | Versioning configured but no active dataserv container with ILM configured |

Ensure dataserv containers are running, enable the S3 lifecycle task manager, and add a cleanup rule:
weka dataservice s3-lifecycle-task enable
weka s3 bucket lifecycle-rule add \ \ --noncurrent

| MINOR | | S3VersioningNoNoncurrentExpirationRule | Versioning configured but no NoncurrentVersionExpiration rule |

Add a cleanup rule with:
weka s3 bucket lifecycle-rule add \ \ --noncurrent

| MINOR | | SSDCapacityTooHigh | Available capacity cannot be fully utilized | To optimize the SSD capacity usage, contact the Customer Success Team. | WARNING | | SystemDefinedTLS | TLS certificate is not user-defined | Replace the auto-generated self-signed certificate with a user-defined certificate by running the `weka security tls set` command. | WARNING | | SystemdSafeShutdownConfigMismatch | Systemd safe shutdown configuration differs across cluster servers | Enable systemd safe shutdown on all servers using `weka local enable-safe-shutdown`. | WARNING | | TLSCertificateExpired | TLS certificate expired | Replace the existing certificate using the `weka security tls set` command. | MAJOR | | TLSCertificateExpiresSoon | TLS certificate is about to expire | Replace the existing certificate using the `weka security tls set` command. | MAJOR | | TelemetryStatusFault | Telemetry status is not streaming | Check your telemetry sinks configuration. | WARNING | | TieredFilesystemOverfillingSSD | Tiered filesystem SSD capacity threshold exceeded | Verify tiering service connectivity to object storage and check for network or authentication issues blocking data migration. Consider increasing tiering upload bandwidth or reviewing tiering policies and thresholds if consistently near capacity. | WARNING | | UserQuotasHardLimitReached | User quota hard limit exceeded | Run the `weka fs quota list --type user` cluster command to get the list of users exceeding their hard quota limits. Clear some space for these users or increase their hard quota limit. | WARNING | | UserQuotasSoftLimitReached | User quota soft limit exceeded | Run the `weka fs quota list --type user` cluster command to get the list of users exceeding their soft quota limits. Clear some space for these users or increase their soft quota limit. | WARNING | | WTracerDaemonWriteIOFailures | WTracer daemon write I/O failures detected | Review System Events for details and check for storage, network, or disk issues affecting trace data writes. | WARNING | | WTracerLostTraces | WTracer lost traces detected | Review System Events and check for high system load, memory pressure, or I/O bottlenecks. | MINOR | --- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter: ``` GET https://docs.weka.io/operation-guide/alerts/list-of-alerts-and-corrective-actions.md?ask=&goal= ``` `ask` is the immediate question: it should be specific, self-contained, and written in natural language. `goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.