List of Alerts

This page lists all the alerts generated by the WekaIO system, along with possible actions to take.

Name
Description
Actions
AdminDefault
Password
The admin password is still set to the factory default. 
Change the admin user password to ensure only authorized users can access the cluster.
AgentNotRunning
The WekaIO local control agent is not running on a host.
Restart the agent with service weka-agent start.
BucketHasNoQuorum
Too many compute nodes are down, causing the bucket compute resource to be unavailable.
Check that the compute nodes and their hosts are up and running and fully connected; contact the WekaIO Support Team if issue is not resolved.
BucketUnresponsive
A compute resource has failed, causing system unavailability. 
Check that the compute nodes and their hosts are up and running and fully connected; contact the WekaIO Support Team if issue is not resolved.
ClockSkew
The clock of a host is skewed in relation to the cluster leader, with a time difference more than the permitted maximum of 30 seconds.
Make sure NTP is configured correctly on the hosts and that their dates are synchronized.
CloudHealth
A host cannot upload events to the WekaIO cloud. 
Check the host has Internet connectivity and is connected to the WekaIO cloud as explained in WekaIO Support Cloud section.
ClusterIsUpgrading
Cluster is upgrading.
If the upgrade doesn't finish normally, contact the WekaIO support for assistance.
DataProtection
Some of the system's data is not fully redundant.
Check which node/host/drive is down and act accordingly.
DedicatedWatchdog
A dedicated WekaIO host requires the installation of a  watchdog driver.
Make sure a watchdog is available at /dev/watchdog. For more information, search the WekaIO knowledgebase in the WekaIO support portal.
DriveDown
A drive is not responding.
Contact WekaIO support to check if the drive should be replaced.
DriveEndurancePercentageUsed
Drive exceeding its life expectancy.
It is recommended to replace the drive before it fails.
DriveEnduranceSparesRemaining
Drive internal spares running too low.
It is recommended to replace the drive before it fails.
DriveNeedsPhaseout
A drive has too many errors.
Phase-out the drive and probably replace it.
FilesystemHasToo
ManyFiles
The filesystem storage configuration is not large enough for the size of files and directory entries being stored. 
Increase the max-files for the filesystem; it may be necessary decrease max-files from another filesystem or install more memory.
HangingIOs
Some IOs are hanging on the node acting as a driver/NFS/backend.
Check that the compute nodes and their hosts are up and running, and fully connected. Also check that if a backend object store is configured, it is connected and responsive. Contact the WekaIO Support Team if issue is not resolved.
HighDrivesCapacity
The average capacity of the SSDs is too high. 
Free-up space on the SSDs or add more SSDs to the cluster.
HugePagesAlloc
WekaIO could not allocate Huge Pages on a host, perhaps because of insufficient memory on the host or if memory is fragmented by usage of other processes.
Reboot the host to avoid memory fragmentation and allow WekaIO to run startIO again. If this fails, verify that the host has enough free memory for use by WekaIO or configure WekaIO to use less memory.
IPConflictDetected
An IP conflict has been detected.
Resolve the conflict of the reported IP.
JumboConnectivity
A host cannot send jumbo frames to any of its cluster peers. 
Check the host network settings and the switch to which it is connected, even if WekaIO seems to be functional, since this will improve performance.
LicenseError
A license conflict exists.
Make sure the cluster is using a correct license; that the license has not expired; and that the cluster allocated space does not exceed the license.
MismatchedDriveFailureDomain
Drive failure domain does not match the failure domain of its attached host.
Either connect the mismatched drive to a host with a matching failure domain, or re-provision the drive to erase its failure domain.
Negative
UnprovisionedCapacity
WekaIO capacity usage changes detected due to cluster upgrade.
One or more of the filesystems need to be resized in order to reclaim capacity; contact the WekaIO Support Team.
NetworkInterfaceLinkDown
A Network interface has link down.
Check the connectivity to the down interface and see if there is anything blocking it.
NoClusterLicense
No license is assigned to the cluster.
Obtain and install a license from get.weka.io.
NodeDisconnected

A node is disconnected from the cluster. 
Check network connectivity to make sure the node can communicate with the cluster.
NodeTiering
Connectivity
A node cannot connect to an object store. 
Check connectivity with the object store and make sure the node can communicate with it.
NotEnoughActiveDrives
There are not enough active failure domains.
Check connectivity, host status  and/or replace problematic drives.
NotEnoughAvailable
MemoryForFilesystems
There are not enough working compute nodes in the cluster to store the file and directory entries for all the filesystems in the cluster. 
Either try to decrease the max-files for some of the filesystems or return the dead compute nodes to get their memory back.
NotEnoughConfigured
MemoryForFilesystems
The total configured memory bytes for filesystems is insufficient to store the file and directory entries of the filesystems in the cluster. 
Decrease the max-files for some of the filesystems, decrease their capacity or increase the configured RAM of the cluster backend hosts.
NumaBalancingEnabled
A host has automatic NUMA balancing enabled which can negatively impact performance. 
To disable, run echo > /proc/sys/kernel/numa_balancing on the host.
OfedVersions
A host Mellanox OFED version ID does not match the one used by the WekaIO container.
Install a supported OFED. If the current version needs to be retained or the alert continues after a supported version is installed, contact the WekaIO Support Team.
PartiallyConnectedNode
A node seems to be only partially connected.
Make sure there is no network connectivity issue.

Operation Guide - Previous

Alerts

Next - Operation Guide

Events

Last updated 4 months ago

Name	Description	Actions
AdminDefault Password	The admin password is still set to the factory default.	Change the admin user password to ensure only authorized users can access the cluster.
AgentNotRunning	The WekaIO local control agent is not running on a host.	Restart the agent with `service weka-agent start.`
BucketHasNoQuorum	Too many compute nodes are down, causing the bucket compute resource to be unavailable.	Check that the compute nodes and their hosts are up and running and fully connected; contact the WekaIO Support Team if issue is not resolved.
BucketUnresponsive	A compute resource has failed, causing system unavailability.	Check that the compute nodes and their hosts are up and running and fully connected; contact the WekaIO Support Team if issue is not resolved.
ClockSkew	The clock of a host is skewed in relation to the cluster leader, with a time difference more than the permitted maximum of 30 seconds.	Make sure NTP is configured correctly on the hosts and that their dates are synchronized.
CloudHealth	A host cannot upload events to the WekaIO cloud.	Check the host has Internet connectivity and is connected to the WekaIO cloud as explained in WekaIO Support Cloud section.
ClusterIsUpgrading	Cluster is upgrading.	If the upgrade doesn't finish normally, contact the WekaIO support for assistance.
DataProtection	Some of the system's data is not fully redundant.	Check which node/host/drive is down and act accordingly.
DedicatedWatchdog	A dedicated WekaIO host requires the installation of a watchdog driver.	Make sure a watchdog is available at /dev/watchdog. For more information, search the WekaIO knowledgebase in the WekaIO support portal.
DriveDown	A drive is not responding.	Contact WekaIO support to check if the drive should be replaced.
DriveEndurancePercentageUsed	Drive exceeding its life expectancy.	It is recommended to replace the drive before it fails.
DriveEnduranceSparesRemaining	Drive internal spares running too low.	It is recommended to replace the drive before it fails.
DriveNeedsPhaseout	A drive has too many errors.	Phase-out the drive and probably replace it.
FilesystemHasToo ManyFiles	The filesystem storage configuration is not large enough for the size of files and directory entries being stored.	Increase the max-files for the filesystem; it may be necessary decrease max-files from another filesystem or install more memory.
HangingIOs	Some IOs are hanging on the node acting as a driver/NFS/backend.	Check that the compute nodes and their hosts are up and running, and fully connected. Also check that if a backend object store is configured, it is connected and responsive. Contact the WekaIO Support Team if issue is not resolved.
HighDrivesCapacity	The average capacity of the SSDs is too high.	Free-up space on the SSDs or add more SSDs to the cluster.
HugePagesAlloc	WekaIO could not allocate Huge Pages on a host, perhaps because of insufficient memory on the host or if memory is fragmented by usage of other processes.	Reboot the host to avoid memory fragmentation and allow WekaIO to run startIO again. If this fails, verify that the host has enough free memory for use by WekaIO or configure WekaIO to use less memory.
IPConflictDetected	An IP conflict has been detected.	Resolve the conflict of the reported IP.
JumboConnectivity	A host cannot send jumbo frames to any of its cluster peers.	Check the host network settings and the switch to which it is connected, even if WekaIO seems to be functional, since this will improve performance.
LicenseError	A license conflict exists.	Make sure the cluster is using a correct license; that the license has not expired; and that the cluster allocated space does not exceed the license.
MismatchedDriveFailureDomain	Drive failure domain does not match the failure domain of its attached host.	Either connect the mismatched drive to a host with a matching failure domain, or re-provision the drive to erase its failure domain.
Negative UnprovisionedCapacity	WekaIO capacity usage changes detected due to cluster upgrade.	One or more of the filesystems need to be resized in order to reclaim capacity; contact the WekaIO Support Team.
NetworkInterfaceLinkDown	A Network interface has link down.	Check the connectivity to the down interface and see if there is anything blocking it.
NoClusterLicense	No license is assigned to the cluster.	Obtain and install a license from get.weka.io.
NodeDisconnected	A node is disconnected from the cluster.	Check network connectivity to make sure the node can communicate with the cluster.
NodeTiering Connectivity	A node cannot connect to an object store.	Check connectivity with the object store and make sure the node can communicate with it.
NotEnoughActiveDrives	There are not enough active failure domains.	Check connectivity, host status and/or replace problematic drives.
NotEnoughAvailable MemoryForFilesystems	There are not enough working compute nodes in the cluster to store the file and directory entries for all the filesystems in the cluster.	Either try to decrease the max-files for some of the filesystems or return the dead compute nodes to get their memory back.
NotEnoughConfigured MemoryForFilesystems	The total configured memory bytes for filesystems is insufficient to store the file and directory entries of the filesystems in the cluster.	Decrease the max-files for some of the filesystems, decrease their capacity or increase the configured RAM of the cluster backend hosts.
NumaBalancingEnabled	A host has automatic NUMA balancing enabled which can negatively impact performance.	To disable, run `echo > /proc/sys/kernel/numa_balancing` on the host.
OfedVersions	A host Mellanox OFED version ID does not match the one used by the WekaIO container.	Install a supported OFED. If the current version needs to be retained or the alert continues after a supported version is installed, contact the WekaIO Support Team.
PartiallyConnectedNode	A node seems to be only partially connected.	Make sure there is no network connectivity issue.