List of alerts and corrective actions

Check WEKA system alerts and take necessary actions based on severity and nature.

Alert name	Description	Corrective actions
AdminDefaultPassword	Default admin password in use	Change the admin user password to ensure only authorized users can access the cluster.
AgentNotRunning	The local agent does not run	Restart the local agent on the specified server using the command ‘service weka-agent start’.
ApproachingClientsUnavailability	Approaching connected clients limit	Ensure all backend containers are up or expand the cluster with more backend containers or servers.
ApproachingSystemLimit	Approaching a system limit	Take the necessary steps based on this action to address the alert.
AutoRemoveTimeoutTooLow	Stateless Client auto-remove timeout too low	Remount the host with a higher auto-remove timeout value.
AvailableMemory	Not enough available memory	Check your system.
BackendNumaBalancingEnabled	NUMA balancing is enabled on a backend server	Disable the automatic NUMA balancing by running the command line 'echo 0 > /proc/sys/kernel/numa_balancing' on the backend server.
BackendVersionsMismatch	Backends mismatch cluster version	Upgrade all the backends to match the cluster's version.
BadDisksCapacityRatio	Bad ratio between smallest and biggest drive	Replace drives so that there will be no such big difference in drives' sizes.
BlockedJrpcMethod	JRPC method is blocked	Unblock the JRPC method by running the command 'blocked_jrpc_methods_remove' or 'blocked_jrpc_methods_clear' manhole.
BondInterfaceCompromised	Network high availability interface compromised	Ensure a proper operation of the network configuration, cables, and NICs.
BucketCapacityExhausting	Buckets are nearing exhaustion of their maximum capacity	Consider migration to a cluster with more buckets.
BucketHasNoQuorum	Too many compute processes are down	Ensure the compute processes on the containers {hosts} are up and running and connected. If the issue is not resolved, contact the Customer Success Team.
BucketUnresponsive	Compute resource failure	Check the connectivity and status of the drives of the leader container. and ensure the compute processes are running and connected. If the issue is not resolved, contact the Customer Success Team.
CPUFrequentStarvation	CPU frequent starvation detected in the last minute	Check the logs of the relevant containers for potential hardware or core allocation problems.
CPUStarvation	CPU starvation detected in the last minute	Check the logs of the relevant containers for potential hardware problems. For specific hang addresses, run /weka/weka_addr2line within the reported weka container on the address to convert them into a symbol name Start IO to allow the task to complete aborting.
CWTaskAbortionStuck	CWTask abortion stuck	Start IO to allow the task to complete aborting
ChokingDetected	High congestion level	For more information, see the System congestion topic in the documentation.
ClientVersionsMismatch	Clients mismatch cluster version	Upgrade the clients to the same version as the cluster by running 'weka local upgrade' locally.
ClockSkew	Clock skew on server	Ensure the NTP is configured correctly on the containers and that their clocks are synchronized.
CloudHealth	WEKA Home disconnected	Check that the server has Internet connectivity and is connected to the Weka Home. See the Weka Home - The Weka support cloud topic in the documentation.
CloudStatsError	Statistics upload failed	See the event details in the System Events.
ClusterInitializationError	Cluster initialization error	Search for the underlying problem causing the error and act accordingly to start IO operations. To clear this alert, run 'weka cluster stop-io'.
ClusterIsUpgrading	Cluster is upgrading	If the upgrade doesn't finish successfully, contact the Customer Success Team.
CoreOverlapping	Core Overlapping	Contact the Customer Success Team.
DataIntegrity	Data integrity problem found	Contact the Customer Success Team.
DataProtection	Partial data protection	Check which process, container, or drive is down and act accordingly.
DedicatedWatchdog	A dedicated server requires the installation of a hardware watchdog driver.	Ensure a hardware watchdog driver is available at /dev/watchdog. For details, search the Knowledge Base in the Weka support portal.
DrainingStuck	Stuck draining	Check the host status and logs for more information.
DriveCriticalWarnings	Drive critical warnings	Deactivate the drive using the command 'weka cluster drive deactivate' and replace it.
DriveDown	Drive down	Contact the Customer Success Team to check if the drive requires a replacement.
DriveEndurancePercentageUsed	Drive exceeds its life expectancy	Replace the specified drive before it fails.
DriveEnduranceSparesRemaining	Drive internal spares run too low	Replace the specified drive before it fails.
DriveNVKVRunningLow	Drive nearing exhaustion of internal resource	Contact the Customer Success Team.
DriveNeedsPhaseout	A drive has too many errors	Deactivate the drive using the command 'weka cluster drive deactivate', and probably replace it.
ExceptionsDuringAlertsEvaluation	Exceptions thrown during alerts evaluation	Check Assertion failures event that may reveal the source of the problem. Contact the Customer Success team if help is required.
FaultsEnabled	Faults are enabled	Contact the Customer Success Team.
FilesystemHasTooManyFiles	Insufficient SSD capacity for metadata on filesystem	To address this issue, consider expanding the filesystem size or removing data and directories. If you have previously configured max-files settings, contact the Customer Success Team for assistance.
FilesystemKMSError	Filesystem KMS Error	Review the filesystem's KMS customization and the KMS configuration and connectivity.
FilesystemsThinProvisioningLowSpace	Filesystems thin provisioning low space	Consider adding SSD capacity to this organization containing these filesystems.
FilesystemsThinProvisioningReserveReached	Filesystems thin provisioning capacity reserve reached	You can create a filesystem or expand the filesystem capacity using the reserved capacity.
HangingCacheSync	Cache sync is hanging	Consider using weka debug fs drop-dirty-cache to drop the cache and enable other clients to access the file (unsynchronized writes will be lost).
HangingClusterTasks	Cluster background task progress is hanging	Contact the Customer Success Team.
HangingIos	Some IOs stop responding	Ensure the compute processes are up and running and connected. If a backend object store is configured, ensure it is connected and responsive. If the issue is not resolved, contact the Customer Success Team.
HighDrivesCapacity	SSD capacity overflow	Free up space on the SSDs or add more SSDs to the cluster. To add SSDs, see the Exapnd specific resources of a container topic in the documentation.
HighLevelOfUnreclaimedCapacityInObjectStore	High level of unreclaimed space in an object store
HighSSDToRAMRatio	High SSD to RAM Ratio	Consider increasing RAM cluster-wide or removing unneeded drives to ensure Filesystems(RAID) requirements and lower SSD to RAM ratio.
HotspotInodes	Some files have a long waiting queue for IOs	Contact the Customer Success Team to help with resolution.
IBNotEnhanced	Enhanced IB mode disabled	Contact Customer Success to correct this issue.
ImbalancedCpuUsage	Imbalanced CPU usage detected in cluster processes	Check system configuration: Examine the system configuration for abnormalities that may be causing the CPU usage imbalance.
JumboConnectivity	A container cannot send jumbo frames.	Check the container network settings and the switch to which the container is connected, and ensure that jumbo frames are enabled. This setting improves performance.
KMSError	KMS Error	Review the KMS configuration and connectivity.
LeaderPreparedForUpgrade	Leader prepared for upgrade	After the upgrade, the leader state automatically returns to normal. If this alert persists, contact the Customer Success Team.
LegacyManualOverridesActive	Legacy manual overrides are active	Contact the Customer Success Team.
LicenseError	License error	Ensure the cluster uses the correct license, the license has not expired, and the allocated space does not exceed the license limits.
LocalTLSCertificateExpired	Local TLS certificate expired	Update the local certificate.
LocalTLSCertificateExpiringSoon	Local TLS certificate is expiring soon	Update the local certificate.
LocalTLSConnectivityToNeighbors	Outgoing TLS connectivity to backends is down	Resolve the TLS issue. One potential cause could be errors in the local CA certificate located in /etc/wekaio/certs.
LowDiskSpace	Low disk space	See the event details in the System Events.
ManualOverridesActive	Manual overrides are active	Contact the Customer Success Team.
ManualOverridesForced	Manual overrides are forced	Contact the Customer Success Team.
MismatchedDriveFailureDomain	A drive failure domain does not match the failure domain of its attached container	Do one of the following: a) Connect the mismatched drive to a container with a matching failure domain. b) Re-provision the drive to erase its failure domain.
NegativeUnprovisionedCapacity	Negative unprovisioned capacity	Resize one or more of the filesystems to reclaim capacity. For more information, contact the Customer Success Team.
NetworkInterfaceLinkDown	Network interface link status down	Check the connectivity to the specified network interface. Verify that nothing blocks it.
NfsLocksDisabled	NFS Locks disabled	Configure config fs using weka nfs global-config set --config-fs=.
NfsServiceDownAlert	NFS Service Down	If down services persist, contact the Customer Success Team.
NoCgroupsConfigured	No cgroups configured warnings	Take measures to enable cgroups if possible.
NoClusterLicense	No license assigned	Obtain and install a license from get.weka.io.
NodeBlacklisted	A process cannot rejoin the cluster	To enable the process to rejoin the cluster, whitelist it by running the command ‘weka debug blacklist disable’.
NodeDisconnected	Process disconnected	Check network connectivity to ensure the processes can communicate with the cluster.
NodeNetworkUnstable	A process with an unstable network detected	Ensure proper network connectivity in the cluster. If the problem is not resolved, contact the Customer Success Team.
NodeRDMANotActive	PA process with supported RDMA is Inactive	Ensure that Mellanox OFED version 4.6 or later is installed on the server and that at least one RDMA-capable device exists.
NodeTieringConnectivity	A process cannot Connect to an object store	Check the connectivity with the object store and ensure the process communicates with it.
NonTlsApisAllowed	Non-TLS APIs are allowed	Update TLS strictness to enforce encrypted TLS APIs over HTTP.
NotEnoughActiveDrives	Reduced data protection	Check the connectivity and server status. Replace failed drives and expand the cluster with new failure domains.
NotEnoughMemoryForFilesystemOperation	Insufficient cluster-wide RAM for proper Filesystem's Operation	Increase RAM cluster-wide to meet Filesystems(RAID) requirements for RAM or remove drives contributing to SSD capacity.
NotEnoughSSDCapacity	Some provisioned capacity is unavailable due to failed drives	Check for down drives.
PartialConnectivityTrackingDisabled	Partial connectivity tracking is disabled	To turn on the Grim Reaper, contact the Customer Success Team.
PartiallyConnectedNode	A partially connected process detected	Ensure proper network connectivity in the cluster. If the problem is not resolved, contact the Customer Success Team.
PassedClientsAvailabilityThreshold	Reached connected clients limit	Add more backend containers or servers to the cluster, check whether the backends are down, or disconnect some clients.
PathsDegraded	Degraded Paths	Contact the Customer Success Team to review path connectivity.
PerformanceDegradedLowRAM	Server low RAM	Ensure all the compute processes are up. Add more servers to the cluster or add RAM to the backend servers.
QuotasHardLimitReached	Directory quota hard limit exceeded	Run 'weka fs quota list' to get the list of directories exceeding their hard quota limits. Clear some space for these directories or increase their hard quota limit.
QuotasSoftLimitReached	Directory quota soft limit exceeded	Run 'weka fs quota list' to get the list of directories exceeding their soft quota limits. Clear some space for these directories or increase their soft quota limit.
RAIDCapacityExhaustion	RAID capacity exhaustion	If the situation does not resolve within minutes, contact the Customer Success Team.
RequestedActionFailure	Requested action failure	Check the logs for more information.
ResourcesNotApplied	Resource changes are not applied	Apply the resource changes by running the command 'weka cluster container apply '.
S3EtcdMigrationAlert	S3 etcd migration	Contact Customer Success to migrate this cluster configuration storage from ETCD to the new built-in Weka solution.
SSDCapacityDiscrepancy	Used SSD capacity mismatches the expected range	Monitor the compute processes' stability and contact the Customer Success Team.
SSDCapacityTooHigh	Available capacity cannot be fully utilized	For improved SSD capacity usage, contact the Customer Success Team for assistance.
SystemDefinedTLS	TLS certificate is not user-defined	Replace the auto-generated self-signed certificate with a user-defined certificate by running the command 'weka security tls set'.
TLSCertificateExpired	TLS certificate expired	Replace the existing certificate by running the command 'weka security tls set'.
TLSCertificateExpiresSoon	TLS certificate is about to expire	Replace the existing certificate by running the command 'weka security tls set'.
TelemetryStatusFault	Telemetry status is not streaming	Check your telemetry sinks configuartion.
TieredFilesystemOverfillingSSD	Tiered filesystems' SSD capacity overfilling	Consider expanding the filesystem size or removing data and directories. Identify, resolve connectivity problems with the configured Object Store and increase the upload bandwidth if required.
TraceDumperDown	Trace dumper is down	Contact the Customer Success Team to restart the trace dumper.'
TracesDisabled	Traces are disabled	To turn the cluster traces, run the command 'weka debug traces start'. For more information, see the Traces management topic in the documentation.
TracesFreezePeriodActive	Freeze traces is active	If the problem persists after the case is resolved, contact the Customer Success Team.
UdpModePerformanceWarning	A backend container is configured in UDP mode	If this is a misconfiguration, add network devices to the specified backend container using the command ‘weka cluster container net add’.
UnstableHosts	Host unstable during upgrade	Check the host status and logs for more information.
UnwritableDisksConfigured	A drive is set to unwritable	If the drive remains unwritable after maintenance, contact the Customer Success Team.