Events list

Explore the various events the WEKA system produces, organized according to their respective categories.

Agent

Type

Severity

Description

ContainerStateEvent

INFO

Container state changed.

DataServiceContainerStateEvent

INFO

Data service container state

WCGroupInvalidResourceConfigEvent

WARNING

Container control group has an invalid resource configuration.

WCGroupStateDisabledEvent

MAJOR

Container control group is disabled.

WCGroupStateEnabledEvent

INFO

Container control group is enabled.

WCGroupValidResourceConfigEvent

INFO

Container control group resource configuration is valid.

Alerts

Type

Severity

Description

AlertCleared

DEBUG

The system has {action} an alert

AlertContinuousEvent

DEBUG

System reports continuous active alerts.

AlertMuted

INFO

Alert is muted.

AlertTriggered

DEBUG

The system has {action} an alert

AlertUnmuted

INFO

Alert is unmuted.

Cloud

Type

Severity

Description

ClientsReportingViaBackendsDisabled

DEBUG

Clients reporting to cloud via backends disabled

ClientsReportingViaBackendsEnabled

DEBUG

Clients reporting to cloud via backends enabled

CloudDisabled

INFO

Cloud WEKA Home integration is disabled.

CloudEnabled

INFO

Cloud WEKA Home integration is enabled.

CloudProxyUpdated

INFO

Cloud WEKA Home proxy configuration is updated.

CloudSetUploadRate

INFO

Cloud WEKA Home upload rate limit changed.

CloudStatsErrorClearedEvent

WARNING

Cloud statistics are now written successfully.

CloudStatsErrorEvent

WARNING

Fails to write cloud statistics for upload.

DiagsUploaded

INFO

Diagnostic information uploaded to Cloud WEKA Home.

LowDiskSpaceClearedEvent

WARNING

Host disk space returned to normal levels.

LowDiskSpaceEvent

WARNING

Host is running low on available disk space.

WekaHomeProxyUploaderStartFailed

DEBUG

Failed to start the Weka Home Proxy Uploader Server

Clustering

Type

Severity

Description

AllBucketsResponsive

INFO

All compute resources are responding normally.

BucketRandomizerSwap

INFO

Bucket randomizer swaps agent assignments.

BucketRedist

INFO

Bucket distribution rebalanced across cluster.

BucketRowHasStaleNodeId

INFO

Bucket contains stale process identifier (NodeId).

CatalogClusterEventCreated

INFO

Catalog cluster configuration change

CatalogClusterEventDestroyed

INFO

Catalog cluster configuration change

CatalogClusterEventUpdated

INFO

Catalog cluster configuration change

CatalogConfigUpdated

INFO

Catalog indexing configuration updated

CatalogCoordinatorUpdated

INFO

Catalog coordinator configuration updated

ClientConnected

INFO

Client connected to cluster.

ClientDisconnected

INFO

Client disconnected from cluster.

ClientRemoved

INFO

Disconnected client is being removed from the cluster.

ClientsUnavailable

CRITICAL

Some clients are unavailable due to insufficient backends.

ClockSkewedHostJoin

MINOR

Container cannot join cluster due to clock skew with existing container.

ClusterDeploymentModeChanged

INFO

Cluster deployment mode changed (auto-detected)

ClusteringFailure

MINOR

Container fails to join the cluster.

ClusterInitializationFailed

MAJOR

Cluster initialization process failed.

ClusterInitialized

INFO

Cluster successfully initialized.

ConfigChangeSetsSliderFull

MINOR

Configuration changeset buffer is full while process pulls configuration updates.

ConfigGenerationHasNoFirstChunk

MINOR

Cannot apply partial configuration generation without an initial chunk.

ConfigSnapshotPulled

MINOR

Configuration snapshot retrieved.

DataServiceClusterCreated

INFO

Data service cluster created

DataServiceClusterDestroyed

INFO

Data service cluster destroyed

DataServiceClusterUpdated

INFO

Data service cluster updated

DoubleUnmatchingMachineIdentifier

MAJOR

Multiple containers detected with identical Agent-Machine-ID but different SMBIOS unique identifiers.

GrimReaperFencingNode

MINOR

Partially connected process selected for fencing by Grim Reaper process.

HostActivated

INFO

Host configuration change

HostAdded

INFO

Host configuration change

HostAdding

INFO

Host configuration change

HostDeactivated

INFO

Host configuration change

HostDeactivating

INFO

Host configuration change

HostDrained

INFO

Host configuration change

HostDraining

INFO

Host configuration change

HostRemoved

INFO

Host configuration change

HostRemoving

INFO

Host configuration change

HostRemovingFailed

INFO

Host configuration change

HostRequestedActionTimeout

WARNING

Host requested action timed out.

LeaderChanged

WARNING

Cluster leadership has been modified to a different process.

LeaderSteppingUpAfterUpgrade

INFO

New cluster leader assumes control after upgrade completion.

NodeNetworkUnstable

MAJOR

Process experiences unstable network connectivity.

NodePartiallyConnected

MINOR

Partially connected process removed from cluster.

NodeRejoined

INFO

Process successfully rejoined cluster.

NodesNotInExpectedState

MAJOR

Some processes are not in the expected operational state.

NodeUnreachable

MINOR

Unreachable process removed from cluster.

OperationTookTooLong

WARNING

Cluster operation exceeds expected completion time.

PersistentUnresponsiveBuckets

CRITICAL

Some compute resources are unresponsive for more than {longUnresponsivenessMinutes} minutes.

PreviousCluster

INFO

Container was previously part of a different cluster.

RejoinFailureReport

MINOR

Containers (nodes) failed to rejoin.

UnresponsiveBuckets

MAJOR

Some compute resources are not responding to requests.

WrongConfigSignatureForRaftSnapshot

MINOR

Attempts to load RAFT snapshot with incompatible configuration signature.

WrongSchemaVersionForRaftSnapshot

MINOR

Attempts to load RAFT snapshot with incompatible schema version.

Config

Type

Severity

Description

BlockTaskStateChanged

DEBUG

Block task state changed.

CachedSnapshotIsNewerThanStreamed

INFO

Cached configuration snapshot is newer than streamed version.

CachedSnapshotIsOlderThanStreamed

INFO

Requested configuration snapshot does not exist yet.

CannotAddAnotherContainerWithTheSameMachineId

MAJOR

Cannot add additional container with duplicate server identifier.

CannotAddAnotherFeContainerWithTheSameMachineId

MAJOR

Cannot add additional frontend container with duplicate server identifier.

ClientTargetVersionChange

INFO

Client target version updated.

ConfigAddedKeyManually

INFO

Configuration value manually added by cluster administrator.

ConfigCapabilityFormatChanged!"max_supported_test_format"

INFO

Cluster capability max_supported_test_format has been updated

ConfigOverridden

INFO

Configuration value overridden by cluster administrator.

ConfigOverrideChanged

INFO

Config override changed.

ConfigOverrideDiscarded

WARNING

Configuration override disabled automatically.

ConfigPropagationTookTooLong

MINOR

Configuration propagation takes too long.

ConfigRemovedKeyManually

INFO

Configuration value removed manually by cluster administrator.

ContainerBlacklistToggle

MAJOR

Container is denied or removed from the denylist.

DirectoryQuotasDisabled

INFO

Directory quotas disabled.

DirectoryQuotasEnabled

INFO

Directory quotas enabled.

EncounteredEntryWithNewVersionInRaftLog

MINOR

Encountered RAFT entry in RAFT log with newer version than existing one.

FetchLocalStateNoDrainTooLongReport

MAJOR

Process cannot drain the local-state changes queue of some overlay children.

HostRequestedActionChange

INFO

Container requested action changed.

IOStatusChanged

INFO

IO status changed.

LeaderIterationGapTooLong

MAJOR

Time gap between the end of the previous leader iteration and the start of the current iteration is too long.

LeaderIterationRuntimeTooLong

MAJOR

Leader iteration execution time exceeded the allowed limit.

LeaderStepDownHttpMonitorUnhealthy

MAJOR

Leader step down because the HTTP monitor is unhealthy

LoginBannerCleared

INFO

Login banner cleared.

LoginBannerDisabled

INFO

Login banner disabled.

LoginBannerEnabled

INFO

Login banner enabled.

LoginBannerSet

INFO

Login banner set.

ProcessBlacklistToggle

MAJOR

Process is denied or removed from the denylist.

ProcessLimitExpanded

INFO

Process limit expanded to allow 24 bits of process ID.

QueryRangeErrorBucketNodeNotInConfig

MAJOR

Unknown NodeId in config bucket.

QueryRangeErrorCriticalBucketNodeNotInConfig

MAJOR

Unknown process ID in configuration bucket.

QueryRangeErrorQueriedBucketNotFound

MAJOR

Bucket Has No Quorum Monitor received unknown BucketId.

QueryRangeErrorQueriedDownNodeNotFound

MAJOR

Bucket Has No Quorum Monitor received unknown NodeId.

QueryRangeErrorRangeErrorException

MAJOR

Bucket Has No Quorum Monitor received RangeError exception.

TooManyContainersWithSameMachineId

MAJOR

Too many containers with same machine ID.

UpgradeBlockTaskStartInvoked

DEBUG

Block task upgrade task start invoked.

WrongVersionForRaftSnapshot

MINOR

Attempt to load RAFT snapshot with unsupported version.

Custom

Type

Severity

Description

Custom

INFO

Custom event.

CustomMajor

MAJOR

Custom major event.

Drive

Type

Severity

Description

AioVecMaxLeakedExceded

DEBUG

AioVec {leaked} netbuf leaked.

CorruptedDrive

MAJOR

Drive has a valid header but is corrupted.

DriveAdded

INFO

Drive provisioned.

DriveCorrupted

MAJOR

Drive has a valid header but is corrupted.

DriveDeactivated

INFO

Drive deactivated.

DriveDead

MAJOR

Drive is unresponsive and fails to return IOs for an extended period.

DriveExcessiveErrors

WARNING

Drive has excessive error rate and will be phased out.

DriveFormatUpgraded

INFO

Drive format upgraded.

DriveIdentifyChanged

INFO

Drive location indicator changed.

DriveImmediateShutdown

MAJOR

Drive shut down immediately.

DriveInitFailed

MAJOR

Drive initialization failed.

DriveIoError

MAJOR

Drive has an IO error.

DriveIoErrorBMS

MAJOR

Drive finds IO error in background media scan.

DriveLimitExceeded

WARNING

Attempt to add more drives than the system supports occurred.

DriveMediumError

MINOR

Drive has a medium error.

DriveNotUnderIOMMU

MAJOR

Drive is not under IOMMU, but host IOMMU is enabled.

DriveNvmeErrorLog

WARNING

NVMe drive error log entry.

DriveNvmeSmartChange

MINOR

NVMe drive SMART status changed.

DriveOutOfNvkvChunks

MAJOR

Drive is out of NVKV chunks.

DriveRemoved

INFO

Drive removed.

DriveSignatureUnknown

MINOR

Drive has an unknown signature.

DriveSmartCriticalWarning

MINOR

Drive SMART reports a critical warning and fails immediately.

DriveStateChangesReport

MINOR

Drive state changes.

DriveStuckIOs

MAJOR

IOs are stuck for an extended period.

DriveUnresponsive

MAJOR

Drive is unresponsive and fails to return IOs for an extended period.

DriveWrongFailureDomain

MINOR

Drive attached to container (host) from incorrect failure domain.

MBufPoison

MAJOR

MBuf got poison error.

NvmeBindTimingOut

MAJOR

The NVMe device binding is stuck, and the server requires a power cycle to recover.

SpdkCuseFilterBlockedCommand

WARNING

SPDK CUSE driver blocked command

Events

Type

Severity

Description

CatalogIngestionCompleted

INFO

Catalog ingestion completed

CatalogMetadataDeleteAllTaskCompleted

INFO

Catalog metadata deletion all task completed

CatalogMetadataDeleteAllTaskCreated

INFO

Catalog metadata deletion all task created

CatalogMetadataDeleteTaskCompleted

INFO

Catalog metadata deletion completed

CatalogMetadataDeleteTaskCreated

INFO

Catalog metadata deletion task created

CatalogMetadataDeleteTaskFailed

MAJOR

Catalog metadata deletion failed

DedupEventsDiscarded

WARNING

Deduplicated events discarded.

EventsDedupReport

INFO

Event deduplication ends.

EventsDiscarded

MINOR

Too many events generated in a short period; some discarded and lost.

Example

INFO

Example.

ExampleAggregated

INFO

Example aggregated.

ExampleDebug

DEBUG

Example debug.

TracesDumperDownEvent

MAJOR

Traces Dumper is inactive.

Filesystem

Type

Severity

Description

BackgroundDirDeleteTask

INFO

Background directory deletion task started.

BackgroundDirDeleteTaskProgress

INFO

Background directory deletion task progress

BlockReadFailure

CRITICAL

Fails to read block.

BlockSeekFinished

MAJOR

Block seek finished.

BlockSeekStarted

MAJOR

Block seek started for secondary metadata block that could not be read.

BrokenFile

MAJOR

File metadata corruption.

CacheFlushHanging

MAJOR

Host hangs while syncing the file's write cache to the cluster.

CatalogTaskAborted

INFO

Catalog task aborted

CheckingTimeoutForFSCK

MAJOR

During integrity check, a timeout occurred; some checks may have been skipped

ChecksumErrorInBackgroundWrite

MAJOR

Checksum error detected by the drive process during block commit.

ChecksumErrorInCommit

MAJOR

Checksum error detected by SSD node in committing block.

ChecksumErrorInWrite

CRITICAL

Checksum error detected by COMPUTE process in write.

ChokingBlockingIOs

MAJOR

Backend choking is now blocking IOs.

ClusterwideJobAborting

INFO

Clusterwide Job configuration change

ClusterwideJobAdded

INFO

Clusterwide Job configuration change

ClusterwideJobRemoved

INFO

Clusterwide Job configuration change

CWTaskTemplateFinished

INFO

Cluster-wide task (CWTask) template finished.

DefaultDirectoryQuotaSet

INFO

Default directory quota set.

DefaultDirectoryQuotaUnset

INFO

Default directory quota unset.

DestageBlocked

CRITICAL

Cannot start destage of bucket.

DestageBlockedByOperation

MAJOR

Mutating operation stalls and prevents destage progress.

DestageHanging

CRITICAL

Destage of a bucket is hanging.

DirectoryQuotaSet

INFO

Directory quota set.

DirectoryQuotaUnset

INFO

Directory quota unset.

DumpSnapHashCompleted

INFO

Finishes snap hash manifest scan.

FailedToAddClusterwideJob

MAJOR

Failed to add the cluster-wide job because the queue has reached its maximum limit

FailedToSplitSliceNoRetry

CRITICAL

Fails to split directory slice; will not retry.

FilesystemAdded

INFO

Filesystem configuration change

FilesystemDeleted

INFO

Filesystem configuration change

FilesystemDownloadStarted

INFO

Filesystem download started.

FilesystemGroupAdded

INFO

Filesystem group configuration change

FilesystemGroupDeleted

INFO

Filesystem group configuration change

FilesystemGroupUpdated

INFO

Filesystem group configuration change

FilesystemMounted

INFO

Mount operation completed successfully.

FilesystemRemoved

INFO

Filesystem configuration change

FilesystemUmounted

INFO

Umount operation completed successfully

FilesystemUpdated

INFO

Filesystem configuration change

FlockSyncHanging

CRITICAL

Bucket cannot finish flock resync with all clients.

ForcedBucketStepdown

MINOR

Bucket forced to step down.

FreeBlockStillUsed

CRITICAL

Finds block falsely marked as free.

FsCapacityLimitReached

WARNING

Filesystem capacity limit reached.

HangingBackendIosDetected

CRITICAL

Some IOs are hanging.

HangingBackendIosNoLongerDetected

INFO

IOs are no longer hanging.

HangingBucketStepDown

WARNING

Bucket step-down is hanging.

HangingDirectorySplit

MAJOR

Directory split makes no progress for a long time.

HangingDriverFrontendIosDetected

CRITICAL

Some IOs are hanging.

HangingDriverFrontendIosNoLongerDetected

INFO

IOs are no longer hanging.

HangingNFSFrontendIosDetected

CRITICAL

Some IOs are hanging.

HangingNFSFrontendIosNoLongerDetected

INFO

IOs are no longer hanging.

IntegrityCheckFinished

DEBUG

Integrity check finished.

IntegrityCheckIssueCritical

CRITICAL

Finds data integrity issue (Critical).

IntegrityCheckIssueCriticalNoDedup

CRITICAL

Finds data integrity issue (Critical).

IntegrityCheckIssueDebug

DEBUG

Finds data integrity issue (Debug).

IntegrityCheckIssueDebugNoDedup

DEBUG

Finds data integrity issue (Debug).

IntegrityCheckIssueMajor

MAJOR

Finds data integrity issue (Major).

IntegrityCheckIssueMajorNoDedup

MAJOR

Finds data integrity issue (Major).

IntegrityCheckIssueMinor

MINOR

Finds data integrity issue (Minor).

IntegrityCheckIssueMinorNoDedup

MINOR

Finds data integrity issue (Minor).

IntegrityCheckIssueWarning

WARNING

Finds data integrity issue (Warning).

IntegrityCheckIssueWarningNoDedup

WARNING

Finds data integrity issue (Warning).

IntegrityCheckStarted

DEBUG

Integrity check started.

IntegrityCheckTransientIssue

DEBUG

Finds transient state expected to be encountered.

IntegrityCheckTransientIssueNoDedup

DEBUG

Finds transient state expected to be encountered.

ManualOverrideStall

WARNING

Service is manually overridden and stalled.

MetadataCommitQueueHang

MINOR

Bucket step down due to hanging metadata commit queue.

ObjectStoreAttachedToFilesystem

INFO

Object store attached to filesystem.

ObjectStoreFinishedDetachingFromFilesystem

INFO

Object store finishes detaching from filesystem.

ObjectStoreStartedDetachingFromFilesystem

INFO

Object store has started detaching from filesystem.

QuotaGraceExpired

WARNING

Directory soft capacity quota reached and grace period expired.

QuotaHardLimitReached

WARNING

Directory hard capacity quota reached.

RAIDMDReadFailureInSnaphashDump

WARNING

Fails to read metadata block from RAID when dumping snapshot manifest.

RemoteClusterAdded

INFO

Remote cluster configuration change

RemoteClusterRemoved

INFO

Remote cluster configuration change

RemoteClusterUpdated

INFO

Remote cluster configuration change

SnapshotContentCopied

INFO

Snapshot content copied.

SnapshotCreated

INFO

Snapshot created.

SnapshotCreationFailed

MAJOR

Snapshot creation fails.

SnapshotDeleted

INFO

Snapshot deleted.

SnapshotDownloadStarted

INFO

Snapshot download started.

SnapshotFilesystemRestored

INFO

Filesystem restored from snapshot.

SnapshotParamsUpdated

INFO

Snapshot updated.

SnapshotPolicyAdded

INFO

Snapshot policy configuration change

SnapshotPolicyAttached

INFO

Snapshot policy attached to filesystem.

SnapshotPolicyDeleted

INFO

Snapshot policy configuration change

SnapshotPolicyDetached

INFO

Snapshot policy detached from filesystem.

SnapshotPolicyUpdated

INFO

Snapshot policy configuration change

SnapshotReplicaReceiveAborted

INFO

Aborted receiving snapshot replica

SnapshotReplicaReceiveFinished

INFO

Finished receiving snapshot replica

SnapshotReplicaReceiveStarted

INFO

Started receiving snapshot replica

SnapshotReplicaSendAborted

INFO

Aborted sending snapshot replica

SnapshotReplicaSendFinished

INFO

Finished sending snapshot replica

SnapshotReplicaSendStarted

INFO

Started sending snapshot replica

SnapshotUploadFailed

MAJOR

Snapshot upload fails.

SnapshotUploadFinished

INFO

Snapshot upload finished.

SnapshotUploadStarted

INFO

Snapshot upload started.

SquelchBlockIdSetAbortedFlushed

DEBUG

While setting the squelch block's block ID for upgrade, it is already changed to invalid.

SquelchBlockIdSetAbortedRewritten

DEBUG

While setting the squelch block's block ID for upgrade, it is already rewritten.

SuperblockUnreadable

CRITICAL

Bucket superblock cannot be loaded from storage.

UnflushedOpOnDeletingSnapview

MAJOR

Unflushed IO operation detected on snapshot being deleted.

InterfaceGroup

Type

Severity

Description

FloatingIpAcquired

INFO

Floating IP address acquired by process.

FloatingIpReleased

INFO

Floating IP address released by process.

FloatingIpRemoveStateTimedout

WARNING

Timeout occurs while removing the floating IP address.

InterfaceGroupAdded

INFO

Interface group configuration change

InterfaceGroupDeleted

INFO

Interface group configuration change

InterfaceGroupIpsAdded

INFO

Interface group IPs configuration change

InterfaceGroupIpsDeleted

INFO

Interface group IPs configuration change

InterfaceGroupPortAdded

INFO

Interface group port configuration change

InterfaceGroupPortDeleted

INFO

Interface group port configuration change

InterfaceGroupUpdated

INFO

Interface group configuration change

IO

Type

Severity

Description

SystemDriveIsTooSlow

MAJOR

System drive response times exceed acceptable thresholds.

KDriver

Type

Severity

Description

DriverAlert

WARNING

Kernel driver generates alert condition.

Kms

Type

Severity

Description

KmsConfigurationAdded

INFO

KMS configuration change

KmsConfigurationRemoved

INFO

KMS configuration change

KmsConfigurationUpdated

INFO

KMS configuration change

Licensing

Type

Severity

Description

LicensingReset

INFO

Licensing state reset to default.

LicensingWorkerStarted

DEBUG

Licensing background worker process starts.

NewLicenseInstalled

INFO

New license installed on the cluster.

PaygLicensingEnabled

INFO

Pay-as-you-go licensing model enabled.

ManualOverride

Type

Severity

Description

ManualOverrideChanged

INFO

Manual configuration override modified.

Network

Type

Severity

Description

ArpServerDuplicateIPDetected

CRITICAL

The ARP server detects a duplicate IP address on the network.

ArpServerFailedToApplyRule

CRITICAL

ARP server fails to apply configuration rules.

ArpServerFailedToInitialize

CRITICAL

ARP server fails to initialize.

ClientNodeDisconnected

INFO

Client disconnects from cluster.

CloudMoveIpFail

MINOR

Fails to reassign IP address in cloud environment.

DefaultDataNetworkingChange

INFO

Default data networking configuration modified.

DpdkIBQkeyMismatch

MAJOR

DPDK InfiniBand queue key mismatch detected.

DpdkInitFailed

MINOR

DPDK initialization failed.

DpdkIovaModeNotDetermined

MAJOR

DPDK unable to determine common IOVA mode across devices.

DpdkPoolSummary

DEBUG

DPDK memory pool status summary.

FipIsNoLongerOnDevice

MAJOR

Floating IP address no longer assigned to expected device.

HangingRPCs

MAJOR

Remote procedure calls exceed expected completion time.

HugepagesAllocationFailure

MINOR

Fails to allocate hugepages memory.

IONodeCannotFetchConfig

WARNING

IO node unable to retrieve cluster configuration and join cluster.

L6PacketFormatNotInSync

MAJOR

Layer 6 packet format inconsistent with process limit flags.

MemoryAllocFailed

MINOR

Memory allocation request fails.

MemoryClaimFailed

MINOR

Memory claim operation fails.

MemoryMigratedAfterPin

MAJOR

Hugepage memory mapping moves after being pinned to physical memory.

MemoryMigratedBeforePin

MINOR

Hugepage memory mapping moves before being pinned to physical memory.

MemoryPinningIoctlFailed

MINOR

Memory pinning system call fails.

MgmtNodeCannotFetchConfig

WARNING

Management process unable to retrieve cluster configuration and join cluster.

NetDeviceLinkDown

MINOR

Network interface link status is DOWN.

NetDeviceLinkUp

MINOR

Network interface link status is UP.

NetSlaveDeviceLinkDown

MINOR

Bonded network slave interface link status is DOWN.

NetSlaveDeviceLinkUp

MINOR

Bonded network slave interface link status is UP.

NetworkBan

MAJOR

Network peer banned due to connectivity issues.

NetworkPortConfigFail

MINOR

Network port configuration operation fails.

NetworkPortDead

MAJOR

Network port does not transmit or receive packets for an extended period.

NetworkUnban

INFO

Network peer unbanned and connectivity restored.

NICNotFound

INFO

Network interface card not found during initialization.

NoConnectivityToLivingNode

MAJOR

Process loses connectivity to all active cluster members.

NodeCannotJoinCluster

WARNING

Process unable to join cluster within expected timeframe.

NodeCannotSendJumboFrames

MINOR

Process unable to send jumbo frames (packets larger than standard MTU).

NodeDisconnected

MINOR

Process disconnected from cluster.

NoJumboFrames

MINOR

Network configuration does not support jumbo frames (packets larger than standard MTU).

RDMAClientDisabled

MINOR

RDMA client optimization disabled.

RDMAClientEnabled

MINOR

RDMA client optimization enabled.

RDMADegraded

MINOR

RDMA performance degraded

RDMADeviceDead

MAJOR

RDMA device does not receive completion notifications for an extended period.

NFS

Type

Severity

Description

NfsAclConfigurationChangeEvent

INFO

NFS ACL configuration changed.

NfsAuthTypeChangeEvent

INFO

NFS Authentication Types configuration changed.

NfsClientGroupAdded

INFO

NFS client group configuration change

NfsClientGroupDeleted

INFO

NFS client group configuration change

NfsClientGroupRuleAdded

INFO

NFS client group rule configuration change

NfsClientGroupRuleDeleted

INFO

NFS client group rule configuration change

NfsClusterStatusActiveEvent

INFO

NFS Cluster is active.

NfsClusterStatusInactiveEvent

CRITICAL

NFS Cluster is inactive.

NfsCustomOptionsUpdated

INFO

NFS custom options configuration change

NfsDirectIOConfigurationChangeEvent

INFO

NFS DirectIO configuration changed.

NfsExportsPermissionsAdded

INFO

NFS export permissions for configuration change

NfsExportsPermissionsDeleted

INFO

NFS export permissions for configuration change

NfsExportsPermissionsUpdated

INFO

NFS export permissions for configuration change

NfsKerberosSetupEvent

INFO

NFS Kerberos Service setup complete.

NfsLdapSetupEvent

INFO

NFS LDAP setup complete.

NfsLocksConfigurationChangeEvent

INFO

NFS Locks configuration changed.

NfsMountFail

WARNING

NFS mount request fails.

NfsPortmapFail

MAJOR

NFS server fails to register in portmap.

NfsServiceDown

CRITICAL

NFS Service is down.

NfsStatsConfigurationChangeEvent

INFO

NFS Extended stats configuration changed.

NfswServiceCriticalEvent

CRITICAL

NFS-W posts event.

NfswServiceInfoEvent

INFO

NFS-W posts event.

NfswServiceWarningEvent

WARNING

NFS-W posts event.

Node

Type

Severity

Description

AssertionFailed

MAJOR

Assertion failed.

GCCrashReport

MINOR

Node has crashed in GC on the previous run

MemoryAllocationFailed

MAJOR

Memory allocation failed.

NodeAbruptExitReport

MINOR

Node has crashed on the previous run

NodeExceptionExit

MAJOR

Process exits with exception.

NodeKernelStack

WARNING

Kernel stack of node before reset.

NodeSignalExit

MAJOR

Process exits due to receiving a signal.

NodeStarted

INFO

Process starts.

NodeStopped

INFO

Process stops.

NodeTraceback

WARNING

Traceback of process before reset.

SoftAssertionFailed

MAJOR

Assertion failed (soft).

ObjectStorage

Type

Severity

Description

ChecksumErrorInDownloadedObject

MINOR

Checksum error detected by compute process in downloaded OBS data block.

ChecksumErrorInReplicatedDataBlock

MAJOR

Checksum error detected by COMPUTE node in a replicated data block

ChecksumErrorOnObjectUpload

MAJOR

Checksum error detected by COMPUTE node when uploading an OBS data block (corrupted after verifying data read from the drive)

DataBlobDownloadFailed

WARNING

Fails downloading data blob header.

DownloadedExtentHasInvalidBlobId

MAJOR

Downloaded extent has invalid blob ID.

DownloadedExtentMissingExpectedBlock

MAJOR

Downloaded extent missing expected block.

ExtentHasFakeRetentionTag

MAJOR

Extent has non-local tag but contains disk-only blocks.

InvalidDataBlobHeader

MAJOR

Invalid header detected by COMPUTE process in downloaded OBS data blob.

ObjectStoreBucketAdded

INFO

Object store bucket configuration change

ObjectStoreBucketDeleted

INFO

Object store bucket configuration change

ObjectStoreBucketUpdated

INFO

Object store bucket configuration change

ObjectStoreGroupAdded

INFO

Object store configuration change

ObjectStoreGroupDeleted

INFO

Object store configuration change

ObjectStoreGroupUpdated

INFO

Object store configuration change

ObjectStoreHasHighLevelOfUnreclaimedCapacity

WARNING

Object store has high level of unreclaimed capacity.

ObjectStoreIsFull

CRITICAL

Object store is full.

ObjectStoreNoLongerHasHighLevelOfUnreclaimedCapacity

INFO

Object store no longer has a high level of unreclaimed capacity.

ObjectStoreStatusDown

MAJOR

Object store status is down.

ObjectStoreStatusUp

INFO

Object store status is UP.

ObsIsMissingObject

CRITICAL

Permanently fails to download object from object storage; object not found.

PersistentChecksumErrorInDownloadedObject

MAJOR

Checksum error detected by compute process in downloaded OBS data block.

Org

Type

Severity

Description

OrgCreated

INFO

Organization created.

OrgDeleted

INFO

Organization deleted.

OrgRenamed

INFO

Organization renamed.

OrgSsdQuotaChanged

INFO

Organization's SSD quota changed.

OrgTotalQuotaChanged

INFO

Organization's total quota changed.

Raft

Type

Severity

Description

IndexChangeDuringStream

INFO

RAFT index changes occur during streaming of the RAFT snapshot.

OutOfMemoryErrorOnLeadershipAgent

MAJOR

Out of memory error on leadership agent.

Raid

Type

Severity

Description

BitmapChecksumMismatch

MINOR

Bitmap checksum mismatch detected.

DataGenerationNumberBug

WARNING

The system encountered an issue while advancing the applied data generation number report from a bucket.

DataProtectionLevelDecreased

MINOR

Data protection level decreased.

DataProtectionLevelIncreased

INFO

Data protection level increased.

DiskNvkvHighUtilization

WARNING

Disk's internal resource (NVKV) has high utilization.

DiskWritableStateChange

INFO

Disk's writable state changed.

DrivesProcessConnectionLost

MINOR

Disk connection lost.

DrivesProcessConnectionRecovered

INFO

Disks quick recovery from lost connection detected.

EnoughActiveFailureDomains

MINOR

Sufficient active failure domains.

EvictionOfPlacementFailed

MAJOR

Eviction of placement encounters a potentially corrupt block marked as used.

FailedRecoveringData

MAJOR

Detects unexpected data; not enough redundant copies available to recover.

FoundCorruptedBlockInStripe

CRITICAL

Detects a corrupt block in a RAID stripe.

FoundOrphanedRaidBlock

MINOR

Detects block marked as Used in RAID stripe yet Free on FS.

HotSpareFailureDomainsUpdated

INFO

Hot spare failure domains updated.

IncorrectScannedBlockChecksum

CRITICAL

Detects a used block with a mismatching checksum.

InFlightCorruptionDetectedByScrubber

MINOR

Detects in-flight corrupt read result from drive.

NoDataProtection

MAJOR

No data protection.

PersistentNoDataProtection

CRITICAL

Cluster is without data protection for a significant period.

PersistentTooManyFailures

CRITICAL

Cluster experiences excessive failures accessing drives for an extended period.

QuorumGenerationNumberBug

WARNING

The system encountered an issue while advancing the applied quorum generation number report from a bucket.

RaidReadFreeBlock

CRITICAL

RAID reads block marked as free.

RaidScrubbingRateUpdated

INFO

RAID scrubber limit updated.

RaidSpaceFull

MINOR

RAID space is full.

RaidsStarted

INFO

RAIDs started.

RepairedCorruptDataFromDrive

CRITICAL

Detected corrupt data from drive[s]. The system will rewrite with the correct data.

ScrubberHanging

MAJOR

RAID Scrubber is hanging on the placement

SingleHopReadCorruptionDetected

MINOR

Single-hop read corruption detected.

SwitchPlacementHanging

MINOR

Active placement to write to is not available due to capacity constraints or disk failures.

TooFewActiveFailureDomains

MAJOR

Too few active failure domains.

TooManyFailures

MAJOR

Too many failures; some data is unavailable.

UsedSSDCapacityCriticalOverflow

CRITICAL

SSD capacity usage critically overflowing; internal spares running out.

UsedSSDCapacityNoLongerOverflows

INFO

SSD capacity usage no longer overflowing.

UsedSSDCapacityOverflow

MAJOR

SSD capacity usage overflowing; internal capacity spares are used.

Resources

Type

Severity

Description

APIServerStarted

INFO

API server started successfully.

APIServerStartFailed

WARNING

Fails to start API server.

BandwidthSelected

INFO

Bandwidth set for host.

CoreAllocated

INFO

Core allocated.

DeviceIsNotAValidNetworkDevice

WARNING

Device is not a valid network device.

DisabledNumaBalancing

INFO

Disabled NUMA Balancing.

DriverLoaded

INFO

Driver attached.

FailedToLoadDriver

WARNING

Fails to load WekaFS driver.

HangingHTTPRequest

MAJOR

Hanging HTTP request detected.

HttpServerFibersExhausted

MAJOR

Hanging HTTP requests exhaust all available fibers.

HugepagesAllocated

INFO

Hugepages allocated.

HugepagesAllocationRetries

WARNING

Hugepages allocation retried.

HugepagesAllocationStarted

INFO

Hugepages allocation started.

HugepagesAllocationTookTooLong

WARNING

Hugepages allocation takes unexpectedly long duration.

InactiveHostCannotJoinCluster

INFO

Inactive host cannot join cluster.

LoadingStableResourcesFailed

INFO

Fails loading stable resources.

NetBufsExhausted

MAJOR

Netbufs are exhausted.

NetDevDriverReloadFailed

MINOR

Net device driver reload failed

NetworkDeviceAllocated

INFO

Network device allocated.

NetworkDeviceHasNoIp

MAJOR

Network device has no IP address.

NetworkDeviceNotUsedByAnySlots

MINOR

Network device not used by any slots.

NoIPsConfiguredForHostJoinWithNoDefaultNet

WARNING

No IP configured for process {nid} with no default-net.

RDMADeviceAllocated

INFO

Allocated dedicated RDMA device

RevertToStableResources

INFO

Reverts to stable resources.

UnlimitedBandwidthSelected

INFO

Bandwidth set to unlimited.

WCGroupContainerEvent

MAJOR

Container Status.

WCGroupUsageMajorEvent

MAJOR

Container {resource} status.

WCGroupUsageWarningEvent

WARNING

Container {resource} status.

S3

Type

Severity

Description

S3AddBucketILMRuleEvent

INFO

A new ILM rule is added to an S3 bucket.

S3AfterUpgradeResult

INFO

S3 after upgrade result.

S3AfterUpgradeStart

INFO

S3 after upgrade started.

S3AsssumeRoleEvent

INFO

An STS token is issued for temporary S3 access.

S3AttachIAMPolicyEvent

INFO

An IAM policy is attached to an S3 bucket.

S3AuditWebhookDisabledEvent

INFO

The S3 audit webhook is disabled.

S3AuditWebhookEnabledEvent

INFO

The S3 audit webhook is enabled.

S3BucketNotificationAdded

INFO

S3 bucket notification added.

S3BucketNotificationRemoved

INFO

S3 bucket notification removed.

S3BucketPerformanceConfigReset

INFO

S3 bucket performance configuration reset

S3BucketPerformanceConfigSet

INFO

S3 bucket performance configuration set

S3BucketTypeChangedEvent

INFO

The S3 bucket type changed

S3ClusterCreated

INFO

A new S3 cluster is successfully created.

S3ClusterDestroyed

INFO

The S3 cluster is successfully destroyed.

S3ClusterDestroyFailed

MAJOR

The attempt to destroy the S3 cluster fails.

S3ClusterPerformanceConfigReset

INFO

S3 cluster performance configuration reset

S3ClusterStatusActiveEvent

INFO

The S3 cluster returns to an operational state.

S3ClusterStatusInactiveEvent

CRITICAL

The S3 cluster is currently non-operational.

S3ClusterUpdated

INFO

Configuration of the S3 cluster is updated.

S3ContainerStateChangesEvent

INFO

The status of an S3 container changes.

S3ContainerStatusActiveEvent

INFO

The S3 container returns to an operational state.

S3ContainerStatusInactiveEvent

MAJOR

The S3 container is currently non-operational.

S3ContainerStatusOnlineEvent

INFO

The S3 container returns to an online state.

S3ContainerStatusSaturatedEvent

MAJOR

The S3 container reaches saturation or capacity limits.

S3CreateBucketEvent

INFO

A new S3 bucket is successfully created.

S3CreateIAMPolicyEvent

INFO

A new IAM policy is created within S3.

S3CreateServiceAccountEvent

INFO

A new S3 service account is created.

S3DestroyBucketEvent

INFO

The S3 bucket is deleted, and all contents are removed.

S3DetachIAMPolicyEvent

INFO

An IAM policy is detached from an S3 role.

S3DrainEvent

INFO

The S3 container is drained for maintenance or shutdown.

S3GlobalPerformanceConfigSet

INFO

S3 global performance configuration set

S3ILMTaskProgress

INFO

S3 Lifecycle task progress.

S3KVAddedEvent

INFO

S3 Config Add Key.

S3KVRemovedEvent

INFO

S3 configuration remove key.

S3KVResetEvent

INFO

S3 KV store configuration reset.

S3KwasFallback

DEBUG

S3 KWAS fallback occurred.

S3MigrationAttachEvent

INFO

S3 attach migration.

S3MigrationDetachEvent

INFO

S3 detach migration.

S3MigrationUpdateEvent

INFO

S3 update migration.

S3MultipleContainersStatusInactiveEvent

CRITICAL

Multiple S3 containers become inactive.

S3NotificationDropped

CRITICAL

Some S3 notifications are lost or dropped.

S3NotificationTargetAdded

INFO

S3 notification-target added.

S3NotificationTargetCertificateAdded

INFO

S3 notification-target certificate added.

S3NotificationTargetCertificateRemoved

INFO

S3 notification-target certificate removed.

S3NotificationTargetRemoved

INFO

S3 notification-target removed.

S3NotificationTargetUpdated

INFO

S3 notification-target updated.

S3NotSupportedInMixedVersion

INFO

Not supported on mixed version.

S3RemoveBucketILMRuleEvent

INFO

An ILM rule is removed from an S3 bucket.

S3RemoveIAMPolicyEvent

INFO

An IAM policy is deleted from the system.

S3RemoveServiceAccountEvent

INFO

An S3 service account is removed.

S3ResetBucketILMRuleEvent

INFO

All ILM rules for the S3 bucket are removed.

S3SetBucketPolicyEvent

INFO

A bucket policy is set or updated for an S3 bucket.

SLBContainerStatusActiveEvent

INFO

The SLB container is active and operational.

SLBContainerStatusInactiveEvent

MAJOR

The SLB container is inactive or stopped.

Security

Type

Severity

Description

BackendJoinTLSVerificationFailure

MINOR

Backend attempts to join cluster but fails TLS verification check.

BackendNodeJoinSecurityPolicyDenied

WARNING

Backend join denied due to security policy.

CaCertSet

INFO

CA certificate added to cluster.

CaCertUnset

INFO

CA certificate unset.

ClientNodeJoinSecurityPolicyDenied

WARNING

Client join denied due to security policy.

ContainerJoinSecretDenied

WARNING

Container fails to rejoin due to missing or incorrect join-secret.

FileSystemSecurityPoliciesChange

INFO

Filesystem security policies changed.

JoinSecurityPoliciesUpdated

INFO

Join Security Policies for {mode} configuration change

LocalTLSCertAdded

INFO

CertificateChange configuration change

LocalTLSCertRemoved

INFO

CertificateChange configuration change

LocalTLSCertUpdated

INFO

CertificateChange configuration change

MountAccessDenied

WARNING

Mount access denied

MountSecurityPolicyAllowed

INFO

Mount access allowed by security policy.

MountSecurityPolicyDenied

WARNING

Mount Access denied by security policy.

OrgSecurityPoliciesChanged

INFO

Organization security policies updated.

OrgTokensRevoked

INFO

Organization API tokens revoked.

SecurityPolicyAccessDenied

WARNING

Access denied due to security policy.

SecurityPolicyCreated

INFO

Security policy configuration change

SecurityPolicyDeleted

INFO

Security policy configuration change

SecurityPolicyUpdated

INFO

Security policy configuration change

TLSSet

INFO

TLS set.

TLSStrictnessUpdated

INFO

TLS strictness updated.

TLSUnset

INFO

TLS unset.

TokenTimeoutsChange

INFO

Token timeouts for authentication updated.

Smb

Type

Severity

Description

SmbAdJoined

INFO

Active Directory configuration change

SmbAdLeft

INFO

Active Directory configuration change

SmbClusterConfigured

INFO

SMB cluster configuration change

SmbClusterCreateCreated

INFO

SMB cluster configuration change

SmbClusterDestroyed

INFO

SMB cluster configuration change

SmbConfigGenerationUpdated

INFO

SMB Config configuration change

SmbShareAdded

INFO

Share configuration change

SmbShareConfigured

INFO

Share configuration change

SmbShareHostnameACERemovedRemoved

INFO

SambaHostnameACE configuration change

SmbShareHostnameACEResetDestroyed

INFO

SambaHostnameACE configuration change

SmbShareRemoved

INFO

Share configuration change

SmbTrustedDomainAdded

INFO

TrustedDomain configuration change

SmbTrustedDomainRemoved

INFO

TrustedDomain configuration change

Statistics

Type

Severity

Description

StatLimitExceeded

WARNING

Set limit on statistics exceeded.

System

Type

Severity

Description

BlockTaskAborted

INFO

Bucket task aborted successfully.

BlockTaskComplete

INFO

Bucket task completes successfully.

BucketsCreated

INFO

System creates buckets.

ClusterTaskAborted

INFO

Cluster task aborted

ClusterTaskAborting

INFO

Cluster task started aborting

ClusterTaskPaused

INFO

Cluster task paused

ClusterTaskResumed

INFO

Cluster task resumed

ClusterTasksCpuLimitUpdated

INFO

Cluster tasks CPU limit set.

ClusterwideTaskChanged

DEBUG

Cluster-wide task changed.

DataServiceLogWriterFlushFailed

MAJOR

Data service log writer flush failed

DataServiceTaskFailedWithError

MINOR

Data service task fails.

DataServiceTaskWaitingForChildTask

INFO

Data service task waiting.

DSAsyncDeleteConfigUpdated

INFO

Data service async-delete config changed

DsShardViewChanged

INFO

Change in data service shard view.

HardwareCriticalLog

MINOR

Hardware Critical Log

HardwareInfoLog

INFO

Hardware Info Log

HardwareWarningLog

WARNING

Hardware Warning Log

HaveEnoughSSDCapacity

MINOR

Sufficient SSD capacity available for all provisioned filesystems.

HostScanDrivesFailed

MAJOR

Call to host_scan_drives failed.

IOStarted

INFO

System starts.

IOStopped

INFO

System stops.

LeaderFailedToReachBaseline

CRITICAL

Reports from too many processes fail to reach the newly elected cluster leader in time.

NotEnoughSSDCapacity

CRITICAL

Insufficient SSD capacity for all provisioned filesystems.

OOMLog

MAJOR

OOM killer has been triggered

QOSConfigReset

INFO

QoS configuration reset

QOSConfigSet

INFO

QoS configuration set

S3ILMTaskManagerConfigUpdated

INFO

S3 ILM task manager config changed.

SleepyDeusExFiber

MAJOR

Deus-ex worker fiber is sleepy; deadlock likely.

StartIORequested

INFO

User requests to start IO.

StopIORequested

INFO

User requests to stop IO.

SystemInfoReport

INFO

Management process starts, reporting OS info.

ThreadPoolCanNotStartThread

MINOR

Reactor's thread pool fails to start a thread.

TooManyFibers

MINOR

Too many fiber allocations.

Telemetry

Type

Severity

Description

AuditTracesStatusChange

INFO

Audit traces status changed.

TelemetryExportAdded

INFO

Telemetry Export configuration change

TelemetryExportDisabled

INFO

Telemetry export disabled.

TelemetryExportEnabled

INFO

Telemetry export enabled.

TelemetryExportRemoved

INFO

Telemetry Export configuration change

TelemetryExportUpdated

INFO

Telemetry Export configuration change

TelemetrySourcesAttachedToExport

INFO

Telemetry Sources Attached To Export

TelemetrySourcesDetachedFromExport

INFO

Telemetry sources detached from export.

Traces

Type

Severity

Description

RemoteTracesDisabled

INFO

Remote traces disabled.

RemoteTracesEnabled

INFO

Remote traces enabled.

RemoteTraceStreamerEndpointInvalid

INFO

Invalid remote trace streamer endpoint.

TracesConfigurationActivated

INFO

Traces configuration change

TracesConfigurationDeactivated

INFO

Traces configuration change

TracesConfigurationReset

INFO

Traces configuration change

TracesConfigurationUpdated

INFO

Traces configuration change

TracesFreezeOnEventOfInterest

MAJOR

Traces frozen on event of interest.

TracesFreezePeriodReset

INFO

Traces freeze period reset.

TracesFreezePeriodSet

INFO

Traces freeze period set.

TracesVerbosityLevelChange

MAJOR

The Traces Verbosity level has been modified

Upgrade

Type

Severity

Description

CleanupUpgradePhaseSkipped

MAJOR

Skips cleanup in upgrade phase: {currentPhase}.

ClientUpgradeRequested

INFO

Client upgrade requested.

ComputeUpgradeFinished

INFO

Compute containers upgrade is finished.

ComputeUpgradeInvoked

INFO

Compute containers upgrade invoked.

ComputeUpgradeStarted

INFO

Compute containers upgrade started.

ContainerUpgradeFailed

MAJOR

Container upgrade fails.

ContainerUpgradeFinished

INFO

Container upgrade finished.

ContainerUpgradeStarted

INFO

Container upgrade started.

DataservUpgradeFinished

INFO

Dataserv containers upgrade finished.

DataservUpgradeStarted

INFO

Dataserv containers upgrade started.

DrivesUpgradeFinished

INFO

Drives containers upgrade finished.

DrivesUpgradeStarted

INFO

Drives containers upgrade started.

EventGeneratorTestWithReport

INFO

An event for the Event Generator Utility, with reporting

ExternalUpgradeCancelled

INFO

External upgrade cancelled.

ExternalUpgradeFinished

INFO

External upgrade finished.

ExternalUpgradeStarting

INFO

External upgrade started.

FailureDomainUpgradeStarted

INFO

Starts to upgrade containers of the failure domain.

FinishedExternalHostUpgrade

INFO

External server upgrade finished.

FrontendUpgradeFinished

INFO

Frontend containers upgrade finished.

FrontendUpgradeStarted

INFO

Frontend containers upgrade started.

StartingExternalHostUpgrade

INFO

External server upgrade started.

TargetVersionChange

DEBUG

Target version changed.

UpgradePaused

INFO

Upgrade paused.

UpgradeResumed

INFO

Upgrade resumed.

UpgradeStateChange

DEBUG

Upgrade state changed.

UpgradeStatusChange

DEBUG

Upgrade status has changed.

WekaVersionDowngraded

WARNING

The cluster is running a version that is lower than the current one.

User

Type

Severity

Description

GuiIdleTimeoutChanged

INFO

GUI idle timeout changed

LDAPAuthDisabled

INFO

LDAP authentication disabled

LDAPAuthEnabled

INFO

LDAP authentication enabled

LDAPConfigUpdated

INFO

LDAP configuration updated

LdapS3AuthFailed

MAJOR

LDAP S3 authentication failed

LdapS3UserCreated

INFO

LDAP S3 user created

LdapS3UserDeleted

INFO

LDAP S3 user deleted

LdapS3UserUpdated

INFO

LDAP S3 user updated

UserCreated

INFO

The user is created.

UserDeleted

INFO

The user is deleted.

UserLoggedIn

INFO

User logged in

UserLoginFailed

INFO

User login failed

UserLoginLocked

MINOR

User login locked

UserPasswordChanged

INFO

The user changed the password.

UserPasswordChangedByAnotherUser

INFO

The administrator changed the user password.

UserRoleChanged

INFO

The user role is changed.