List of Alerts

This page lists all the alerts generated by the Weka system, along with possible actions to take.

Name

Description

Actions

AdminDefault Password

The admin password is still set to the factory default.

Change the admin user password to ensure only authorized users can access the cluster.

AgentNotRunning

The Weka local control agent is not running on a host.

Restart the agent with service weka-agent start.

AutoRemoveTimeoutTooLow

Stateless Client auto-remove timeout too low.

Remount the host with a higher auto-remove timeout value.

BackendNumaBalancingEnabled

A host has automatic NUMA balancing enabled which can negatively impact performance.

To disable, run echo 0 > /proc/sys/kernel/numa_balancing on the backend host.

BucketHasNoQuorum

Too many compute nodes are down, causing the bucket compute resource to be unavailable.

Check that the compute nodes and their hosts are up and running and fully connected; Contact the Weka Support Team if the issue is not resolved.

BucketUnresponsive

A compute resource has failed, causing system unavailability.

Check that the compute nodes and their hosts are up and running and fully connected; Contact the Weka Support Team if the issue is not resolved.

ChokingDetected

High congestion level detected in the cluster.

For more information, refer to System Congestion.

ClientNumaBalancingEnabled

A host has automatic NUMA balancing enabled which can negatively impact performance.

To disable, run echo 0 > /proc/sys/kernel/numa_balancing on the client host.

ClockSkew

The clock of a host is skewed in relation to the cluster leader, with a time difference more than the permitted maximum of 30 seconds.

Make sure NTP is configured correctly on the hosts and that their dates are synchronized.

CloudHealth

A host cannot upload events to the Weka cloud.

Check the host has Internet connectivity and is connected to the Weka cloud as explained in the Weka Support Cloud section.

ClusterIsUpgrading

Cluster is upgrading.

If the upgrade doesn't finish normally, contact the Weka support for assistance.

DataProtection

Some of the system's data is not fully redundant.

Check which node/host/drive is down and act accordingly.

DedicatedWatchdog

A dedicated Weka host requires the installation of a watchdog driver.

Make sure a watchdog is available at /dev/watchdog. For more information, search the Weka knowledgebase in the Weka support portal.

DriveDown

A drive is not responding.

Contact the Weka support team to check if the drive should be replaced.

DriveEndurancePercentageUsed

Drive exceeding its life expectancy.

It is recommended to replace the drive before it fails.

DriveEnduranceSparesRemaining

Drive internal spares running too low.

It is recommended to replace the drive before it fails.

DriveNeedsPhaseout

A drive has too many errors.

Phase-out the drive and probably replace it.

DriveScanError

Drive scan errors

Check if the drive is still present and OK

FilesystemHasToo ManyFiles

The filesystem storage configuration for the size of file and directory entries is exceeding (or about to exceed).

Increase the max-files for the filesystem.

HangingIOs

Some IOs are hanging on the node acting as a driver/NFS/backend.

Check that the compute nodes and their hosts are up and running, and fully connected. Also check that if a backend object store is configured, it is connected and responsive. Contact the Weka Support Team if the issue is not resolved.

HighDrivesCapacity

The average capacity of the SSDs is too high.

Free-up space on the SSDs or add more SSDs to the cluster.

IPConflictDetected

An IP conflict has been detected.

Resolve the conflict of the reported IP.

JumboConnectivity

A host cannot send jumbo frames to any of its cluster peers.

Check the host network settings and the switch to which it is connected, even if Weka seems to be functional since this will improve performance.

KmsConfigurationError

KMS Configuration Error

Review the KMS credentials, permissions, and configuration, as suggested in KMS management.

LicenseError

A license conflict exists.

Make sure the cluster is using a correct license, the license has not expired, and the cluster allocated space does not exceed the license.

MismatchedDriveFailureDomain

The drive failure domain does not match the failure domain of its attached host.

Either connect the mismatched drive to a host with a matching failure domain, or re-provision the drive to erase its failure domain.

NegativeUnprovisionedCapacity

Weka capacity usage changes detected due to cluster upgrade.

One or more of the filesystems need to be resized in order to reclaim capacity. Contact the Weka Support Team.

NetworkInterfaceLinkDown

A Network interface has a link down status.

Check the connectivity to the down interface and see if there is anything blocking it.

NoClusterLicense

No license is assigned to the cluster.

Obtain and install a license from get.weka.io.

NodeBlacklisted

There is a blacklisted node in the cluster.

Use weka debug blacklist disable to whitelist nodes so they can rejoin the cluster.

NodeDisconnected

A node is disconnected from the cluster.

Check network connectivity to make sure the node can communicate with the cluster.

NodeNetworkUnstable

A node seems to have an unstable network. As a consequence, it has been fenced by the system and does not contribute resources to the Weka cluster.

Make sure there is no network connectivity issue in the cluster. Contact the Weka Support Team if the issue is not resolved.

NodeRDMANotActive

RDMA is supported on the host but it is inactive.

Make sure Mellanox OFED version 4.6 or higher is properly installed on the host.

NodeTieringConnectivity

A node cannot connect to an object-store.

Check connectivity with the object store and make sure the node can communicate with it.

NotEnoughActiveDrives

There are not enough active failure domains.

Check connectivity, host status, and/or replace problematic drives.

OfedVersions

A host Mellanox OFED version ID does not match the one used by the Weka container.

Install a supported OFED. If the current version needs to be retained or the alert continues after a supported version is installed, contact the Weka Support Team.

PartiallyConnectedNode

A node seems to be only partially connected.

Make sure there is no network connectivity issue. Contact the Weka Support Team if the issue is not resolved.

PerformanceDegradedLowRAM

The host is running low on RAM. Additional Metadata entries are swapped to the SSD. This might impact performance.

Make sure all the compute hosts and processes are up, add more hosts to the Weka cluster, or the configured RAM of the cluster backend hosts.

ResourcesNotApplied

There are changes to host resources that are not applied in the Weka cluster.

To apply changes run weka cluster host apply <host_id>

UdpModePerformanceWarning

The backend host is configured in UDP mode.

If this is a misconfiguration use weka cluster host net addto add network devices to this host.