List of alerts and corrective actions
Explore the list of system alerts, their meanings, and the recommended corrective actions. Understanding these alerts helps you respond to system events effectively.
This list covers alerts of WARNING severity and above. Alerts with DEBUG severity are informational and are not shown by default; to see them on a cluster, run weka alerts --severity DEBUG.
For detailed maintenance and troubleshooting articles, search the WEKA Knowledge Base in the WEKA support portal. Access to the portal requires a login and is available for registered customers and internal users. You can also contact the Customer Success Team.
AdminDefaultPassword
Default admin password in use
Change the admin password to restrict access to authorized users.
WARNING
ApproachingSystemLimit
Approaching a system limit
Follow the in-screen actions.
MAJOR
AsymmetricPeerMTU
A process has peers with asymmetric MTU
Verify that switch ports, bonded interfaces, and NIC MTU settings are consistent across the path between this process and its peers.
WARNING
AutoRemoveTimeoutTooLow
Stateless client auto-remove timeout too low
Remount the host with a higher auto-remove timeout value.
WARNING
AvailableMemory
Not enough available memory
Check the system memory usage.
MAJOR
BackendNumaBalancingEnabled
NUMA balancing is enabled on a backend server
Disable automatic NUMA balancing on the backend server: echo 0 > /proc/sys/kernel/numa_balancing.
WARNING
BackendVersionsMismatch
Backends mismatch cluster version
Upgrade all backends to match the cluster version.
WARNING
BadDisksCapacityRatio
Large discrepancy between the smallest and largest drive
Replace drives with comparable capacities to minimize the differences between the drives' sizes.
MAJOR
BondInterfaceCompromised
Network high availability interface compromised
Verify the network configuration, cables, and NICs.
MINOR
BondMTUMismatch
Bond has inconsistent MTU across slaves
Inspect the host's interfaces with ip link show and align the MTU of all bond slaves with the bond master.
WARNING
BucketUnresponsive
Compute resource failure
Check connectivity and drive status on the leader container. Ensure compute processes are running. If the issue persists, contact the Customer Success Team.
CRITICAL
BucketsImbalance
FS buckets are not properly balanced between processes
If this alert persists for a long time, contact the Customer Success Team for assistance.
WARNING
CatalogIndexFsCapacityCritical
Catalog index filesystem is critically low on space
Increase the capacity of the catalog index filesystem to resume catalog ingestion.
CRITICAL
CatalogServiceDegraded
Catalog service is degraded
Check the status of catalog service containers and ensure they are online with healthy frontend processes.
MAJOR
CatalogServiceInactive
Catalog service is inactive
Restore catalog service containers to bring the service back to active state.
CRITICAL
ClockSkew
Clock skew on server
Ensure NTP is configured correctly on the containers and that their clocks are synchronized.
MINOR
CloudHealth
WEKA Home disconnected
Check that the server has internet connectivity and is connected to WEKA Home. See the "WEKA Home - The WEKA support cloud" topic.
MINOR
ClusterInitializationError
Cluster initialization error
Search for the underlying problem causing the error and act accordingly to start IO operations. To clear this alert, run the weka cluster stop-io command.
WARNING
CompressedSpaceFragmented
Compressed free space is too fragmented
Check whether DATA_DEFRAG tasks are paused or hanging. If this persists, contact the Customer Success Team.
WARNING
CoreOverlapping
Core overlapping
Contact the Customer Success Team.
MAJOR
DataIntegrity
Data integrity problem found
The filesystem scan has identified data integrity problems that affect data accessibility or correctness. Contact the Customer Success Team.
CRITICAL
DataProtection
Partial data protection
Restore full data protection immediately. If the cluster remains resilient to one failure, identify the down process, container, or drive and resolve the issue.
MINOR
DataServiceConfigurationNotSet
Dataservice global configuration filesystem is not set
Set the config_fs for dataservice by running the weka dataservice global-config set --config-fs command.
MINOR
DrainingStuck
Host stuck in draining state
Check the host status and logs to identify blockers such as slow data migration, capacity constraints, or I/O errors. Verify cluster health and available capacity, then contact the Customer Success Team if the issue persists.
MINOR
DriveAutoEjected
Drive automatically ejected
The drive was automatically ejected due to persistent issues. Monitor the drive phase-out progress and consider planning a replacement.
WARNING
DriveAutoReactivating
Drive auto-reactivation
Monitor the drive status.
WARNING
DriveCriticalWarnings
Drive critical warnings
Deactivate the drive using the weka cluster drive deactivate command and replace it.
MAJOR
DriveDown
Drive failed
Contact the Customer Success Team to assess drive health and determine if physical replacement is required. Provide the drive UUID, error message, and host details for faster diagnosis.
MINOR
DriveEndurancePercentageUsed
Drive exceeds its life expectancy
Replace the specified drive before it fails.
MAJOR
DriveEnduranceSparesRemaining
Drive internal spares run too low
Replace the specified drive before it fails.
MAJOR
DriveHasPartialFailureReports
Some compute processes are failing to access disk
Monitor the drive closely. If the issue persists, consider deactivating it using weka cluster drive deactivate.
MAJOR
DriveNVKVRangesInsufficient
Drive has insufficient NVKV ranges
Contact the Customer Success Team.
MAJOR
DriveNeedsPhaseout
A drive has too many errors
Deactivate the affected drive using the weka cluster drive deactivate <drive-id> command; add --hard if automatic ejection failed. Consider replacing the drive.
MAJOR
DriveUnwritableTooLong
Drive has been unwritable for too long
A drive that has been unwritable for a long time risks NVKV overload when it becomes writable again. Workaround: fully deactivate the drive, wait for it to reach the INACTIVE state, mark it writable, then activate it again. If the issue persists, contact the Customer Success Team.
MAJOR
FilesystemsThinProvisioningLowSpace
Filesystems thin provisioning low space
Consider adding more SSD capacity to the tenant containing these filesystems.
WARNING
FsMetadataInconsistency
Filesystem metadata inconsistency found
The filesystem scan identified metadata inconsistencies, including anomalies in block metadata, snapshot structures, or upgrade-related data. These do not immediately affect data reads, but if left unresolved, they may impact filesystem reliability. Contact the Customer Success Team.
MAJOR
FsScanIssue
Filesystem scan internal error detected
The filesystem scan encountered internal errors during validation, indicating a problem with the scan process rather than with the underlying data. Rerun the scan. If errors persist, contact the Customer Success Team.
WARNING
FsScanReclaimableSpace
Reclaimable storage space detected
The filesystem scan identified orphaned blocks or leaked storage space. These issues do not affect data integrity or accessibility, but represent space that can be reclaimed. They are typically caused by interrupted operations. Schedule a maintenance window to reclaim the affected space. If you need guidance, contact the Customer Success Team.
MINOR
GroupQuotasHardLimitReached
Group quota hard limit exceeded
Run the weka fs quota list --type group cluster command to get the list of groups exceeding their hard quota limits. Clear some space for these groups or increase their hard quota limit.
WARNING
GroupQuotasSoftLimitReached
Group quota soft limit exceeded
Run the weka fs quota list --type group cluster command to get the list of groups exceeding their soft quota limits. Clear some space for these groups or increase their soft quota limit.
WARNING
HAInterfaceMTUMismatch
HA group has inconsistent MTU across interfaces
Inspect the host's DPDK interfaces and align the MTU of all interfaces of the same link layer.
WARNING
HangingCacheSync
Cache sync is hanging
Consider using weka debug fs drop-dirty-cache to drop the cache and enable other clients to access the file (unsynchronized writes will be lost).
MINOR
HardwareComponentFaulty
Hardware component fault detected
Check the management controller (BMC or IPMI) for diagnostic details and take corrective action.
MAJOR
HardwareComponentWarning
Hardware component warning
Check the management controller (BMC or IPMI) for details on the affected components.
WARNING
HardwareMonitoringUnavailable
Hardware monitor unavailable
Verify that BMC or IPMI settings are configured correctly in the container resources. If the issue persists, contact Customer Success.
MINOR
HighDrivesCapacity
SSD capacity overflow
Free up space on the SSDs or add more SSDs to the cluster. See the "Expand specific resources of a container" topic.
MAJOR
HighSSDToRAMRatio
High SSD to RAM ratio
Consider increasing RAM cluster-wide, or removing unneeded drives, to meet filesystem (RAID) requirements and lower the SSD to RAM ratio.
WARNING
JumboConnectivity
A container cannot send jumbo frames
Check the container's network settings and the switch it is connected to, and ensure jumbo frames are enabled. This setting improves performance.
WARNING
KMSError
KMS error
Review the KMS configuration and connectivity.
MAJOR
LicenseError
License error
Ensure the cluster uses the correct license, that the license has not expired, and that the allocated space does not exceed the license limits.
The LicenseError alert includes a 5% grace threshold. It is triggered only when used capacity exceeds 105% of the licensed limit, which allows for minor overages without generating an immediate alert.
WARNING
LowDiskSpace
Low disk space
See the event details in the System Events.
MINOR
MismatchedDriveFailureDomain
A drive's failure domain does not match the failure domain of the container it is attached to
Attach the drive to a container with a matching failure domain, or re-provision the drive to clear its failure-domain assignment.
MAJOR
MismatchedJoinSecrets
Backend containers do not have the same join secrets
This may create problems rejoining or reforming the cluster. Ensure all backend containers have the same join secrets.
WARNING
NetworkInterfaceLinkDown
Network interface link status down
Check the connectivity to the specified network interface and verify that nothing is blocking it.
MINOR
NetworkSpaceApplyStuck
Network Space apply stuck
Inspect with weka debug netspace apply-status. If a backend is permanently down, remove it from the cluster, or force the operation to terminal with weka debug netspace force-complete.
MAJOR
NfsLocksDisabled
NFS locks disabled
Configure config_fs using the weka nfs global-config set --config-fs=<filesystem name> command.
WARNING
NfsServiceDownAlert
NFS service down
If down services persist, contact the Customer Success Team.
MAJOR
NicPciSettingsMisconfigured
NIC PCI firmware settings suboptimal
One or more NIC firmware settings, ADVANCED_PCI_SETTINGS or PCI_WR_ORDERING, are not set to 1. These settings improve PCIe performance for high-throughput workloads. Applying them requires resetting the NIC firmware. ADVANCED_PCI_SETTINGS applies to the whole card. PCI_WR_ORDERING applies per port, so run the commands for each PCI address listed in the alert.
On the affected server, stop the WEKA containers, then run:
weka local run mstconfig -d <pci-address> set ADVANCED_PCI_SETTINGS=1 PCI_WR_ORDERING=1
weka local run mstfwreset -d <pci-address> reset
Restart the WEKA containers.
WARNING
NoCgroupsConfigured
No cgroups configured warnings
Enable and configure cgroups (v1/v2) by following the Cgroups configuration section in the documentation.
WARNING
NoClusterLicense
No license assigned
Obtain and install a license from get.weka.io.
WARNING
NoHotSpareFailureDomains
No hot-spare failure domains have been configured
Assign one or more hot-spare failure domains by running the weka cluster hot-spare command.
WARNING
NodeDisconnected
Process disconnected
Check the network connectivity to ensure the processes can communicate with the cluster.
MINOR
NodeNetworkUnstable
A process with an unstable network detected
Ensure proper network connectivity in the cluster. If the problem is not resolved, contact the Customer Success Team.
WARNING
NodeTieringConnectivity
A process cannot connect to an object store
A process cannot connect to the object store due to network connectivity issues, process health issues, or the object storage vendor equipment itself. Check the connectivity with the object store and ensure the process can communicate with it.
MAJOR
NotEnoughActiveDrives
Reduced data protection
Check the connectivity and server status, and activate drives in more failure domains.
MAJOR
NotEnoughSSDCapacity
Some provisioned capacity is unavailable due to failed drives
Check for down drives.
MAJOR
NotificationQueueHighLoad
S3 notification queue reached high watermark
Reduce the notification load or increase the queue capacity. If the issue persists, contact the Customer Success Team.
MINOR
NotificationSendFailure
S3 notification send failure
Check system logs for details and restore Kafka service availability.
MAJOR
PartialHugepageAllocation
Not enough memory
Check the system's hugepage and memory configuration.
MAJOR
PartiallyConnectedNode
A partially connected process detected
Ensure proper network connectivity in the cluster. If the issue persists, contact the Customer Success Team.
MINOR
PathsDegraded
Degraded paths
Contact the Customer Success Team to review path connectivity.
MINOR
PerformanceDegradedLowRAM
Low server RAM
Add more servers to the cluster, add RAM to the backend servers, or increase the memory allocation to the compute processes.
MAJOR
QuotasHardLimitReached
Directory quota hard limit exceeded
Run the weka fs quota list cluster command to get the list of directories exceeding their hard quota limits. Clear some space for these directories or increase their hard quota limit.
WARNING
QuotasSoftLimitReached
Directory quota soft limit exceeded
Run the weka fs quota list cluster command to get the list of directories exceeding their soft quota limits. Clear some space for these directories or increase their soft quota limit.
WARNING
RAIDCapacityExhaustion
RAID capacity nearing exhaustion
Monitor the situation for approximately 5 minutes to allow for automatic rebalancing. If RAID capacity usage remains high or continues to increase, contact the Customer Success Team immediately to prevent write failures.
MAJOR
S3LifecycleTaskCannotStart
S3 lifecycle tasks cannot start - missing dataservice containers
Set up dataservice containers by following the "Set up a Data Services container for background tasks" documentation.
MINOR
S3UpgradeInProgress
S3 is undergoing a software upgrade
If the upgrade doesn't finish successfully, contact the Customer Success Team.
WARNING
S3VersioningNoDataservIlm
Versioning configured but no active dataserv container with ILM configured
Ensure dataserv containers are running, enable the S3 lifecycle task manager, and add a cleanup rule:
weka dataservice s3-lifecycle-task enable
weka s3 bucket lifecycle-rule add <bucket> <days> --noncurrent
MINOR
S3VersioningNoNoncurrentExpirationRule
Versioning configured but no NoncurrentVersionExpiration rule
Add a cleanup rule with:
weka s3 bucket lifecycle-rule add <bucket> <days> --noncurrent
MINOR
SSDCapacityTooHigh
Available capacity cannot be fully utilized
To optimize the SSD capacity usage, contact the Customer Success Team.
WARNING
SystemDefinedTLS
TLS certificate is not user-defined
Replace the auto-generated self-signed certificate with a user-defined certificate by running the weka security tls set command.
WARNING
SystemdSafeShutdownConfigMismatch
Systemd safe shutdown configuration differs across cluster servers
Enable systemd safe shutdown on all servers using weka local enable-safe-shutdown.
WARNING
TLSCertificateExpired
TLS certificate expired
Replace the existing certificate using the weka security tls set command.
MAJOR
TLSCertificateExpiresSoon
TLS certificate is about to expire
Replace the existing certificate using the weka security tls set command.
MAJOR
TelemetryStatusFault
Telemetry status is not streaming
Check your telemetry sinks configuration.
WARNING
TieredFilesystemOverfillingSSD
Tiered filesystem SSD capacity threshold exceeded
Verify tiering service connectivity to object storage and check for network or authentication issues blocking data migration. Consider increasing tiering upload bandwidth or reviewing tiering policies and thresholds if consistently near capacity.
WARNING
UserQuotasHardLimitReached
User quota hard limit exceeded
Run the weka fs quota list --type user cluster command to get the list of users exceeding their hard quota limits. Clear some space for these users or increase their hard quota limit.
WARNING
UserQuotasSoftLimitReached
User quota soft limit exceeded
Run the weka fs quota list --type user cluster command to get the list of users exceeding their soft quota limits. Clear some space for these users or increase their soft quota limit.
WARNING
WTracerDaemonWriteIOFailures
WTracer daemon write I/O failures detected
Review System Events for details and check for storage, network, or disk issues affecting trace data writes.
WARNING
WTracerLostTraces
WTracer lost traces detected
Review System Events and check for high system load, memory pressure, or I/O bottlenecks.
MINOR
Last updated