# Events list ### Agent | **Type** | **Severity** | **Description** | | --------------------------------- | ------------ | --------------------------------------- | | ContainerStateEvent | INFO | Container State | | WCGroupInvalidResourceConfigEvent | WARNING | Container CGroup Resource Configuration | | WCGroupStateDisabledEvent | MAJOR | Container CGroup State | | WCGroupStateEnabledEvent | INFO | Container CGroup State | | WCGroupValidResourceConfigEvent | INFO | Container CGroup Resource Configuration | ### Alerts | **Type** | **Severity** | **Description** | | -------------------- | ------------ | ----------------------------------------- | | AlertCleared | DEBUG | The system has {action} an alert | | AlertContinuousEvent | DEBUG | The system has reported continuous alerts | | AlertMuted | INFO | Alert muted | | AlertTriggered | DEBUG | The system has {action} an alert | | AlertUnmuted | INFO | Alert unmuted | ### Cloud | **Type** | **Severity** | **Description** | | --------------------------- | ------------ | ------------------------------------------ | | CloudDisabled | INFO | Cloud disabled | | CloudEnabled | INFO | Cloud enabled | | CloudProxyUpdated | INFO | Cloud proxy updated | | CloudSetUploadRate | INFO | Cloud upload rate changed | | CloudStatsErrorClearedEvent | WARNING | Cloud stats have been written successfully | | CloudStatsErrorEvent | WARNING | Error writing cloud stats for upload | | DiagsUploaded | INFO | Diags uploaded | | LowDiskSpaceClearedEvent | WARNING | The host no longer has low disk space | | LowDiskSpaceEvent | WARNING | The host has low disk space | ### Clustering | **Type** | **Severity** | **Description** | | ----------------------------------------- | ------------ | ---------------------------------------------------------------------------------------------- | | AllBucketsResponsive | INFO | All compute resources are responding. | | BucketRandomizerSwap | INFO | Bucket randomizer performed agents swap | | BucketRedist | INFO | The buckets were redistributed in the cluster | | BucketRowHasStaleNodeId | INFO | Bucket has stale node id | | ClientConnected | INFO | Client connected | | ClientDisconnected | INFO | Client disconnected | | ClientRemoved | INFO | Start removing a disconnected client from the cluster. | | ClientsUnavailable | CRITICAL | Some clients are unavailable because too many backends are down. | | ClockSkewedHostJoin | MINOR | The container (host) cannot join because of a clock skew. | | ClusteringFailure | MINOR | Conainer clustering failed. | | ClusterInitializationFailed | MAJOR | The cluster initialization is failed. | | ClusterInitialized | INFO | The cluster is successfully initialized. | | ClusterLeaderInsufficientPeerConnectivity | INFO | The cluster leader has connectivity issues with too many peers. | | ConfigChangeSetsSliderFull | MINOR | Configuration changeset slider is full while the process (node) is pulling the configuration. | | ConfigGenerationHasNoFirstChunk | MINOR | Applying a partial configuration generation is prohibited. | | ConfigSnapshotPulled | MINOR | Configuration snapshot is pulled. | | DoubleUnmatchingMachineIdentifier | MAJOR | There is a container with same Agent-Machine-ID, but different SMBIOS UID. | | GrimReaperFencingNode | MINOR | A partially connected process (node) is selected to be fenced by the grim-reaper. | | HostActivated | INFO | Host configuration change | | HostAdded | INFO | Host configuration change | | HostAdding | INFO | Host configuration change | | HostDeactivated | INFO | Host configuration change | | HostDeactivating | INFO | Host configuration change | | HostDrained | INFO | Host configuration change | | HostDraining | INFO | Host configuration change | | HostRemoved | INFO | Host configuration change | | HostRemoving | INFO | Host configuration change | | HostRemovingFailed | INFO | Host configuration change | | LeaderChanged | WARNING | The cluster leader has changed. | | LeaderSteppingUpAfterUpgrade | INFO | The cluster leader is stepping up after upgrade | | NodeNetworkUnstable | MAJOR | A process (node) with an unstable network is detected. | | NodePartiallyConnected | MINOR | A partially connected process (node) was removed. | | NodeRejoined | INFO | The process (node) has rejoined the cluster. | | NodesNotInExpectedState | MAJOR | Some processes (nodes) are not in the expected state. | | NodeUnreachable | MINOR | An unreachable process (node) was removed. | | OperationTookTooLong | WARNING | The operation took too long. | | PersistentUnresponsiveBuckets | CRITICAL | Some compute resources are not responding for more than {longUnresponsivenessMinutes} minutes. | | PreviousCluster | INFO | This host was part of another cluster before | | RejoinFailureReport | MINOR | Containers (nodes) failed to rejoin. | | UnresponsiveBuckets | MAJOR | Some compute resources are not responding. | | WrongConfigSignatureForRaftSnapshot | MINOR | Tried to load a RAFT snapshot with an unsupported configuration root snapshot signature. | | WrongSchemaVersionForRaftSnapshot | MINOR | Tried to load a RAFT snapshot with an unsupported schema version. | ### Config | **Type** | **Severity** | **Description** | | ------------------------------------------------------------ | ------------ | -------------------------------------------------------------------------------------------------- | | BlockTaskStateChanged | DEBUG | Block task state changed. | | CachedSnapshotIsNewerThanStreamed | INFO | The streamed snapshot is older than the cached snapshot. | | CachedSnapshotIsOlderThanStreamed | INFO | Requested snapshot doesn't exist yet. | | CannotAddAnotherContainerWithTheSameMachineId | WARNING | Too many containers with the same machine ID. | | ClientTargetVersionChange | INFO | Client target version has been set | | ConfigAddedKeyManually | INFO | A configuration value is added manually by the cluster administrator. | | ConfigCapabilityFormatChanged!"max\_supported\_test\_format" | INFO | Cluster capability max\_supported\_test\_format has been updated | | ConfigOverridden | INFO | A configuration value is overridden by the cluster administrator. | | ConfigOverrideChanged | INFO | A config override is changed. | | ConfigOverrideDiscarded | WARNING | A config override was disabled automatically. | | ConfigPropagationToBackendsDurationCritical | CRITICAL | Config propagation to backend nodes taking too long - beyond critical threshold - aborting leader. | | ConfigPropagationToBackendsTakingTooLong | MINOR | Config propagation to backend nodes taking too long (still ongoing). | | ConfigPropagationToBackendsTookTooLong | MINOR | Config propagation to backend nodes took too long. | | ConfigPropagationToClientsDurationCritical | CRITICAL | Config propagation to client nodes taking too long - beyond critical threshold - aborting leader. | | ConfigPropagationToClientsTakingTooLong | MINOR | Config propagation to client nodes taking too long (still ongoing). | | ConfigPropagationToClientsTookTooLong | MINOR | Config propagation to client nodes took too long. | | ConfigRemovedKeyManually | INFO | A configuration value is removed manually by the cluster administrator. | | ContainerBlacklistToggle | INFO | A container is blacklisted/removedFromBlacklist. | | DirectoryQuotasDisabled | INFO | Directory quotas were disabled | | DirectoryQuotasEnabled | INFO | Directory Quotas were enabled | | EncounteredEntryWithNewVersionInRaftLog | MINOR | Encountered a RAFT entry RAFT log with a newer version than mine. | | FetchLocalStateNoDrainTooLongReport | MAJOR | The node cannot manage to drain the local-state changes queue of some overlay children. | | HostRequestedActionChange | INFO | Host requested action changed. | | IOStatusChanged | INFO | IO status changed. | | LeaderIterationGapTooLong | MAJOR | Time gap between end of previous leader iteration and start of current iteration is too long. | | LeaderIterationRuntimeTooLong | MAJOR | Leader iteration execution time was too long. | | LoginBannerCleared | INFO | Login banner has been cleared | | LoginBannerDisabled | INFO | Login banner disabled | | LoginBannerEnabled | INFO | Login banner enabled | | LoginBannerSet | INFO | Login banner has been set | | ProcessBlacklistToggle | INFO | A process is blacklisted/removedFromBlacklist. | | ProcessLimitExpanded | INFO | Process limit expanded to allow 24-bits of node ID | | QueryRangeErrorBucketNodeNotInConfig | MAJOR | Unknown NodeId in a config bucket (Ref 499822) | | QueryRangeErrorCriticalBucketNodeNotInConfig | MAJOR | Unknown NodeId in config bucket (Ref 499822) | | QueryRangeErrorQueriedBucketNotFound | MAJOR | BucketHasNoQuorumMonitor got an unknown BucketId (Ref 499822) | | QueryRangeErrorQueriedDownNodeNotFound | MAJOR | BucketHasNoQuorumMonitor got an unknown NodeId (Ref 499822) | | QueryRangeErrorRangeErrorException | MAJOR | BucketHasNoQuorumMonitor got a RangeError exception (Ref 499822) | | UpgradeBlockTaskStartInvoked | DEBUG | Block task upgrade task start invoked. | | WrongVersionForRaftSnapshot | MINOR | Tried to load a RAFT snapshot with an unsupported version. | ### Custom | **Type** | **Severity** | **Description** | | ----------- | ------------ | ------------------ | | Custom | INFO | Custom event | | CustomMajor | MAJOR | Custom major event | ### Drive | **Type** | **Severity** | **Description** | | ------------------------- | ------------ | ---------------------------------------------------------------------------------------------- | | CorruptedDrive | MAJOR | A drive has a valid header but it is corrupted. | | DriveAdded | INFO | Drive provisioned | | DriveCorrupted | MAJOR | A drive has a valid header but it is corrupted. | | DriveDeactivated | INFO | Drive deactivated | | DriveDead | MAJOR | The drive is unresponsive and fails to return IOs for an extended period. | | DriveExcessiveErrors | WARNING | The drive has excessive error rate and will be phased out. Contact the Customer Success Team. | | DriveFormatUpgraded | INFO | The drive format was upgraded. | | DriveImmediateShutdown | MAJOR | The drive had to be shutdown immediately. Contact the Customer Success Team. | | DriveInitFailed | MAJOR | A drive failed to initialize. | | DriveIoError | MAJOR | A drive has an IO error. | | DriveIoErrorBMS | MAJOR | A drive found an IO error in a background media scan. | | DriveLimitExceeded | WARNING | An attempt to add more drives than supported. | | DriveMediumError | MINOR | A drive has a medium error. | | DriveNotUnderIOMMU | MAJOR | The drive is not under IOMMU but the host IOMMU is enabled. Contact the Customer Success Team. | | DriveNvmeErrorLog | WARNING | An NVMe drive error log entry. | | DriveNvmeSmartChange | MINOR | The NVMe drive SMART status has changed. | | DriveOutOfNvkvChunks | MAJOR | A drive is out of NVKV chunks | | DriveRemoved | INFO | Drive removed | | DriveSignatureUnknown | MINOR | A drive has an unknown signature. | | DriveSmartCriticalWarning | MINOR | The drive SMART reports a critical warning and therefore fails it immediately. | | DriveStateChangesReport | MINOR | Drive state changes. | | DriveStuckIOs | MAJOR | The IOs are stuck for an extended period. | | DriveUnresponsive | MAJOR | The drive is unresponsive and fails to return IOs for an extended period. | | DriveWrongFailureDomain | MINOR | A drive is attached to a container (host) from an incorrect failure domain. | | MBufPoison | MAJOR | MBUf got poison error. | | NvmeBindTimingOut | MAJOR | The NVMe device binding is stuck and the server needs a power cycle to recover. | ### Events | **Type** | **Severity** | **Description** | | --------------------- | ------------ | ------------------------------------------------------------------------------------------ | | DedupEventsDiscarded | WARNING | Deduplicated events discarded. | | EventsDedupReport | INFO | Event deduplication ended. | | EventsDiscarded | MINOR | Too many events were generated in a short period, so some of them were discarded and lost. | | Example | INFO | Example | | ExampleAggregated | INFO | Example aggregated | | ExampleDebug | DEBUG | Example debug | | TracesDumperDownEvent | MAJOR | The Traces Dumper is inactive | ### Filesystem | **Type** | **Severity** | **Description** | | ------------------------------------------ | ------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------- | | BlockReadFailure | CRITICAL | Failed to read a block | | BlockSeekFinished | MAJOR | Block seek finished | | BlockSeekStarted | MAJOR | Block seek started for a secondary metadata block that could not be read | | BrokenFile | MAJOR | File metadata corruption | | CacheFlushHanging | MAJOR | Host is hanging while trying to sync a file's write cache to the cluster | | CharterNotificationDropped | MAJOR | Dropped a charter notification to an unresponsive client | | CheckingTimeoutForFSCK | MAJOR | During integrity check a timeout occurred; some checks may have been skipped | | ChecksumErrorInBackgroundWrite | MAJOR | Checksum error detected by SSD node in a committing block | | ChecksumErrorInCommit | MAJOR | Checksum error detected by SSD node in a committing block | | ChecksumErrorInWrite | CRITICAL | Checksum error detected by COMPUTE node in a write | | ChokingBlockingIOs | MAJOR | Backend choking is now blocking IOs | | ClusterwideJobAborting | INFO | Clusterwide Job configuration change | | ClusterwideJobAdded | INFO | Clusterwide Job configuration change | | ClusterwideJobRemoved | INFO | Clusterwide Job configuration change | | CWTaskTemplateFinished | INFO | A cluster wide task (CWTask) template finished. | | DefaultDirectoryQuotaSet | INFO | A default directory quota was set | | DefaultDirectoryQuotaUnset | INFO | A default directory quota was unset | | DestageBlocked | CRITICAL | Destage of a bucket cannot start | | DestageBlockedByOperation | MAJOR | A mutating operation is stalling and preventing destage from making progress | | DestageHanging | CRITICAL | Destage of a bucket is hanging | | DirectoryQuotaSet | INFO | The directory quota was set | | DirectoryQuotaUnset | INFO | The directory quota was unset | | DumpSnapHashCompleted | INFO | Finished a snap hash manifest scan | | FailedToAddClusterwideJob | MAJOR | Failed to add clusterwide job because the queue has reached maximum limit | | FailedToSplitSliceNoRetry | CRITICAL | Failed to split a directory slice - wont retry | | FilesystemAdded | INFO | Filesystem configuration change | | FilesystemDeleted | INFO | Filesystem configuration change | | FilesystemDownloadStarted | INFO | Filesystem download started | | FilesystemGroupAdded | INFO | Filesystem group configuration change | | FilesystemGroupDeleted | INFO | Filesystem group configuration change | | FilesystemGroupUpdated | INFO | Filesystem group configuration change | | FilesystemRemoved | INFO | Filesystem configuration change | | FilesystemUpdated | INFO | Filesystem configuration change | | FlockSyncHanging | CRITICAL | A bucket cannot finish flock resync with all the clients | | ForcedBucketStepdown | MINOR | Bucket forced to step down | | FreeBlockStillUsed | CRITICAL | Found a block falsely marked as free | | FsCapacityLimitReached | WARNING | Filesystem capacity limit has been reached | | HangingBackendIosDetected | CRITICAL | Some IOs are hanging. | | HangingBackendIosNoLongerDetected | INFO | IOs are no longer hanging. | | HangingBucketStepDown | WARNING | Bucket step-down is hanging | | HangingDirectorySplit | MAJOR | Directory split hasn't any made progress for a long time | | HangingDriverFrontendIosDetected | CRITICAL | Some IOs are hanging. | | HangingDriverFrontendIosNoLongerDetected | INFO | IOs are no longer hanging. | | HangingNFSFrontendIosDetected | CRITICAL | Some IOs are hanging. | | HangingNFSFrontendIosNoLongerDetected | INFO | IOs are no longer hanging. | | IntegrityCheckFinished | DEBUG | Integrity check has finished. | | IntegrityCheckIssueCritical | CRITICAL | Found a data integrity issue (Critical) | | IntegrityCheckIssueCriticalNoDedup | CRITICAL | Found a data integrity issue (Critical) | | IntegrityCheckIssueDebug | DEBUG | Found a data integrity issue (Debug) | | IntegrityCheckIssueDebugNoDedup | DEBUG | Found a data integrity issue (Debug) | | IntegrityCheckIssueMajor | MAJOR | Found a data integrity issue (Major) | | IntegrityCheckIssueMajorNoDedup | MAJOR | Found a data integrity issue (Major) | | IntegrityCheckIssueMinor | MINOR | Found a data integrity issue (Minor) | | IntegrityCheckIssueMinorNoDedup | MINOR | Found a data integrity issue (Minor) | | IntegrityCheckIssueWarning | WARNING | Found a data integrity issue (Warning) | | IntegrityCheckIssueWarningNoDedup | WARNING | Found a data integrity issue (Warning) | | IntegrityCheckStarted | DEBUG | Integrity check has started. | | IntegrityCheckTransientIssue | DEBUG | Found a transient state which is expected to be encountered. Can be ignored, unless it persists. In which case a non transient issue event will be produced | | IntegrityCheckTransientIssueNoDedup | DEBUG | Found a transient state which is expected to be encountered. Can be ignored, unless it persists. In which case a non transient issue event will be produced | | ManualOverrideStall | WARNING | A service has been manually overridden and stalled. | | MetadataCommitQueueHang | MINOR | Bucket step down due to hanging metadata commit queue | | ObjectStoreAttachedToFilesystem | INFO | The object store is attached to the filesystem. | | ObjectStoreFinishedDetachingFromFilesystem | INFO | The object store finished detaching from the filesystem. | | ObjectStoreStartedDetachingFromFilesystem | INFO | The object store started detaching from the filesystem. | | QuotaGraceExpired | WARNING | Directory soft capacity quota has been reached and grace period expired | | QuotaHardLimitReached | WARNING | Directory hard capacity quota has been reached | | RAIDMDReadFailureInSnaphashDump | WARNING | Failed to read metadata block from RAID when dumping the snapshot manifest | | SnapshotContentCopied | INFO | Snapshot content copied | | SnapshotCreated | INFO | Snapshot created | | SnapshotCreationFailed | MAJOR | Snapshot creation failed | | SnapshotDeleted | INFO | Snapshot deleted | | SnapshotDownloadStarted | INFO | Snapshot download started | | SnapshotFilesystemRestored | INFO | Filesystem restored from snapshot | | SnapshotParamsUpdated | INFO | Snapshot updated | | SnapshotPolicyAdded | INFO | Snapshot policy configuration change | | SnapshotPolicyAttached | INFO | The snapshot policy is attached to the filesystem | | SnapshotPolicyDeleted | INFO | Snapshot policy configuration change | | SnapshotPolicyDetached | INFO | The snapshot policy is detached from the filesystem | | SnapshotPolicyUpdated | INFO | Snapshot policy configuration change | | SnapshotUploadFailed | MAJOR | Snapshot upload failed | | SnapshotUploadFinished | INFO | Snapshot upload finished | | SnapshotUploadStarted | INFO | Snapshot upload started | | SquelchBlockIdSetAbortedFlushed | DEBUG | While setting a squelch block's block id for upgrade was already changed to invalid | | SquelchBlockIdSetAbortedRewritten | WARNING | While setting a squelch block's block id for upgrade was already rewritten to something else | | SuperblockUnreadable | CRITICAL | Superblock of a bucket could not be loaded | | UnflushedOpOnDeletingSnapview | MAJOR | Unflushed IO on a deleting snapshot | ### InterfaceGroup | **Type** | **Severity** | **Description** | | ----------------------------- | ------------ | ------------------------------------------------- | | FloatingIpAcquired | INFO | A floating IP was acquired by the process (node). | | FloatingIpReleased | INFO | A floating IP was released by the process (node). | | FloatingIpRemoveStateTimedout | WARNING | Timeout occurred during floating IP removal. | | InterfaceGroupAdded | INFO | Interface group configuration change | | InterfaceGroupDeleted | INFO | Interface group configuration change | | InterfaceGroupIpsAdded | INFO | Interface group IPs configuration change | | InterfaceGroupIpsDeleted | INFO | Interface group IPs configuration change | | InterfaceGroupPortAdded | INFO | Interface group port configuration change | | InterfaceGroupPortDeleted | INFO | Interface group port configuration change | | InterfaceGroupUpdated | INFO | Interface group configuration change | ### IO | **Type** | **Severity** | **Description** | | -------------------- | ------------ | -------------------------------- | | SystemDriveIsTooSlow | MAJOR | System drive is slow to respond. | ### KDriver | **Type** | **Severity** | **Description** | | ----------- | ------------ | --------------- | | DriverAlert | WARNING | Driver Alert | ### Kms | **Type** | **Severity** | **Description** | | ----------------------- | ------------ | ------------------------ | | KmsConfigurationAdded | INFO | KMS configuration change | | KmsConfigurationRemoved | INFO | KMS configuration change | | KmsConfigurationUpdated | INFO | KMS configuration change | ### Licensing | **Type** | **Severity** | **Description** | | ---------------------- | ------------ | ------------------------------ | | LicensingReset | INFO | Licensing state has been reset | | LicensingWorkerStarted | DEBUG | Licencing worker started | | NewLicenseInstalled | INFO | New license installed | | PaygLicensingEnabled | INFO | PAYG licensing enabled | ### ManualOverride | **Type** | **Severity** | **Description** | | --------------------- | ------------ | ----------------------------- | | ManualOverrideChanged | INFO | A manual override is changed. | ### Network | **Type** | **Severity** | **Description** | | ---------------------------- | ------------ | -------------------------------------------------------------------------------------------- | | ArpServerDuplicateIPDetected | CRITICAL | ARP server duplicate IP detected. | | ArpServerFailedToApplyRule | CRITICAL | ARP server failed to apply rules. | | ArpServerFailedToInitialize | CRITICAL | ARP server failed to initialize. | | ClientNodeDisconnected | WARNING | A client disconnected from cluster. | | CloudMoveIpFail | MINOR | Move IP on cloud failed. | | DefaultDataNetworkingChange | INFO | The default data networking configuration has changed. | | DpdkIBQkeyMismatch | MAJOR | DPDK IB qkey Mismatch | | DpdkInitFailed | MINOR | DPDK initialization failed | | DpdkIovaModeNotDetermined | MAJOR | DPDK is not able to find a common iova-mode | | DpdkPoolSummary | DEBUG | Summary of DPDK pool status | | FipIsNoLongerOnDevice | MAJOR | IP is no longer on device. | | HangingRPCs | MAJOR | Some RPCs are hanging too long. | | HugepagesAllocationFailure | MINOR | Hugepages allocation failure. | | IONodeCannotFetchConfig | WARNING | The IO node cannot join the cluster for too long. | | L6PacketFormatNotInSync | WARNING | L6 packet format not in sync with process limits flags, indicates a non fatal internal error | | MemoryAllocFailed | MINOR | Memory allocation failed | | MemoryClaimFailed | MINOR | Memory claim failed | | MemoryMigratedAfterPin | MAJOR | Hugepage mapping migrated after it was pinned | | MemoryMigratedBeforePin | MINOR | Hugepage mapping migrated before it was pinned | | MemoryPinningIoctlFailed | MINOR | Memory pinning ioctl failed | | MgmtNodeCannotFetchConfig | WARNING | The management process (node) cannot join the cluster for too long. | | NetDeviceLinkDown | MINOR | Network interface DOWN | | NetDeviceLinkUp | MINOR | Network interface UP | | NetSlaveDeviceLinkDown | MINOR | Network slave interface DOWN | | NetSlaveDeviceLinkUp | MINOR | Network slave interface UP | | NetworkBan | MAJOR | Ban Network Peer | | NetworkPortConfigFail | MINOR | Network port configuration failed. | | NetworkPortDead | MAJOR | Network Port hasn't passed packets for a long period of time, it is likely dead | | NetworkUnban | INFO | Unban Network Peer | | NICNotFound | INFO | NIC not found when initializing | | NoConnectivityToLivingNode | MAJOR | A process (node) is disconnected from living peers. | | NodeCannotJoinCluster | WARNING | The process (node) cannot join the cluster for too long. | | NodeCannotSendJumboFrames | MINOR | A process (node) cannot send jumbo packets. | | NodeDisconnected | MINOR | A process (node) disconnected from cluster. | | NoHardwareWatchdog | MAJOR | No hardware watchdog found | | NoJumboFrames | MINOR | The network does not allow jumbo packets through. | | RDMAClientDisabled | MINOR | RDMA optimization disabled | | RDMAClientEnabled | MINOR | RDMA optimization enabled | | RDMADeviceDead | MAJOR | RDMA device did not recieve completions for a long period of time, it is likely dead | ### NFS | **Type** | **Severity** | **Description** | | ----------------------------------- | ------------ | ------------------------------------------------- | | NfsAclConfigurationChangeEvent | INFO | NFS Acl Configuration changed | | NfsAuthTypeChangeEvent | INFO | NFS Authentication Types Configuration | | NfsClientGroupAdded | INFO | NFS client group configuration change | | NfsClientGroupDeleted | INFO | NFS client group configuration change | | NfsClientGroupRuleAdded | INFO | NFS client group rule configuration change | | NfsClientGroupRuleDeleted | INFO | NFS client group rule configuration change | | NfsClusterStatusActiveEvent | INFO | NFS Cluster is active | | NfsClusterStatusInactiveEvent | CRITICAL | NFS Cluster is inactive | | NfsCustomOptionsUpdated | INFO | NFS custom options configuration change | | NfsDirectIOConfigurationChangeEvent | INFO | NFS DirectIO Configuration changed | | NfsExportsPermissionsAdded | INFO | NFS export permissions for configuration change | | NfsExportsPermissionsDeleted | INFO | NFS export permissions for configuration change | | NfsExportsPermissionsUpdated | INFO | NFS export permissions for configuration change | | NfsKerberosSetupEvent | INFO | NFS Kerberos Service setup is done. | | NfsLdapSetupEvent | INFO | NFS LDAP setup is done. | | NfsLocksConfigurationChangeEvent | INFO | NFS Locks Configuration changed | | NfsMountFail | WARNING | NFS mount request failed. | | NfsPortmapFail | MAJOR | The NFS server failed to register in the portmap. | | NfsServiceDown | CRITICAL | Nfs Service Down. | | NfsStatsConfigurationChangeEvent | INFO | NFS Extended stats Configuration changed | ### Node | **Type** | **Severity** | **Description** | | -------------------- | ------------ | --------------------------------------------------- | | AssertionFailed | MAJOR | Assertion failed. | | GCCrashReport | MINOR | Node has crashed in GC on the previous run | | NodeAbruptExitReport | MINOR | Node has crashed on the previous run | | NodeExceptionExit | MAJOR | A process (node) exited with an exception. | | NodeKernelStack | WARNING | Kernel stack of node before reset | | NodeSignalExit | MAJOR | A process (node) existed due to receiving a signal. | | NodeStarted | DEBUG | A process (node) started. | | NodeStopped | DEBUG | A process (node) stopped. | | NodeTraceback | WARNING | Traceback of node before reset | | SoftAssertionFailed | MAJOR | Assertion failed (soft). | ### ObjectStorage | **Type** | **Severity** | **Description** | | ---------------------------------------------------- | ------------ | ----------------------------------------------------------------------------------------------------------------------------- | | ChecksumErrorInDownloadedObject | MINOR | Checksum error detected by COMPUTE node in a downloaded OBS data block | | ChecksumErrorOnObjectUpload | MAJOR | Checksum error detected by COMPUTE node when uploading an OBS data block (corrupted after verifying data read from the drive) | | DataBlobDownloadFailed | WARNING | Failed downloading data blob header | | DownloadedExtentHasInvalidBlobId | MAJOR | Downloaded extent has invalid blob id | | DownloadedExtentMissingExpectedBlock | MAJOR | Downloaded extent missing expected block | | ExtentHasFakeRetentionTag | MAJOR | Extent has non-local tag but has disk-only blocks | | InvalidDataBlobHeader | MAJOR | Invalid header detected by COMPUTE node in a downloaded OBS data blob | | ObjectStoreBucketAdded | INFO | Object store bucket configuration change | | ObjectStoreBucketDeleted | INFO | Object store bucket configuration change | | ObjectStoreBucketUpdated | INFO | Object store bucket configuration change | | ObjectStoreGroupAdded | INFO | Object store configuration change | | ObjectStoreGroupDeleted | INFO | Object store configuration change | | ObjectStoreGroupUpdated | INFO | Object store configuration change | | ObjectStoreHasHighLevelOfUnreclaimedCapacity | WARNING | The object store has a high level of unreclaimed capacity. | | ObjectStoreIsFull | CRITICAL | Object store is full | | ObjectStoreNoLongerHasHighLevelOfUnreclaimedCapacity | INFO | The object store no longer has a high level of unreclaimed capacity. | | ObjectStoreStatusDown | MAJOR | The object store status is down. | | ObjectStoreStatusUp | INFO | The object store status is UP. | | ObsIsMissingObject | CRITICAL | Permanently failed to download an object from object storage - The object was not found | | PersistentChecksumErrorInDownloadedObject | MAJOR | Checksum error detected by COMPUTE node in a downloaded OBS data block | ### Org | **Type** | **Severity** | **Description** | | -------------------- | ------------ | ---------------------------------------- | | OrgCreated | INFO | The organization is created. | | OrgDeleted | INFO | The organization is deleted. | | OrgRenamed | INFO | The organization is renamed. | | OrgSsdQuotaChanged | INFO | The organization SSD quota is changed. | | OrgTotalQuotaChanged | INFO | The organization total quota is changed. | ### Raft | **Type** | **Severity** | **Description** | | --------------------------------- | ------------ | ----------------------------------------------------------------- | | IndexChangeDuringStream | INFO | The RAFT index changed during the streaming of the raft snapshot. | | OutOfMemoryErrorOnLeadershipAgent | CRITICAL | Out of memory error on the leadership agent. | ### Raid | **Type** | **Severity** | **Description** | | ------------------------------------ | ------------ | -------------------------------------------------------------------------------------------------------------------------------------------- | | BitmapChecksumMismatch | MINOR | Bitmap checksum mismatch detected. | | DataGenerationNumberBug | WARNING | An issue with the advancement of the applied data generation number report from a bucket. | | DataProtectionLevelDecreased | MINOR | The data protection level decreased. | | DataProtectionLevelIncreased | INFO | The data protection level increased. | | DiskNvkvHighUtilization | WARNING | Disk's internal resource (NVKV) has high utilization | | DiskWritableStateChange | INFO | Disk's writable state changed | | DrivesProcessConnectionLost | MINOR | Disk connection lost. | | DrivesProcessConnectionRecovered | INFO | Disks quick recovery from lost connection detected. | | EnoughActiveFailureDomains | MINOR | Sufficient active failure domains. | | EvictionOfPlacementFailed | MAJOR | Eviction of placement encountered a potentially corrupt block marked as used. | | FailedRecoveringData | MINOR | Detected unexpected data, not enough redundant copies are available to recover it | | FoundCorruptedBlockInStripe | CRITICAL | Detected corrupt block in a RAID stripe. | | HotSpareFailureDomainsUpdated | INFO | Hot spare failure domains updated. | | IncorrectScannedBlockChecksum | CRITICAL | Detected used block with a mismatching checksum. | | InFlightCorruptionDetectedByScrubber | MINOR | Detected in-flight corrupt read result from drive | | NoDataProtection | MAJOR | No data protection. | | PersistentNoDataProtection | CRITICAL | Cluster has no data protection a significant period of time, the problem is likely not ephemeral. | | PersistentTooManyFailures | CRITICAL | Cluster has been experiencing too many failures when accessing drives for a significant period of time, the problem is likely not ephemeral. | | QuorumGenerationNumberBug | WARNING | An issue with the advancement of the applied quorum generation number report from a bucket. | | RaidReadFreeBlock | MAJOR | RAID had read a block marked as free | | RaidScrubbingRateUpdated | INFO | RAID scrubber limit updated | | RaidSpaceFull | MAJOR | A RAID Space is full. | | RaidsStarted | INFO | RAIDs started. | | RepairedCorruptDataFromDrive | CRITICAL | Detected corrupt data from drive\[s]. The system will rewrite with the correct data. | | ScrubberHanging | MAJOR | RAID Scrubber is hanging on placement | | SingleHopReadCorruptionDetected | MINOR | Single-hop read corruption detected. | | SoulDeserializationFailed | MINOR | Failed to deserialize bucket soul during startup | | SoulSerializationFailed | MINOR | Failed to serialize bucket soul during graceful shutdown | | SoulValidationFailed | INFO | Bucket soul validation failed during startup | | SwitchPlacementHanging | MINOR | An active placement to write to is not available because of capacity constraints or disk failures. | | TooFewActiveFailureDomains | MAJOR | Too few active failure domains. | | TooManyFailures | MAJOR | Too many failures and some of the data is unavailable. | | UsedSSDCapacityCriticalOverflow | CRITICAL | SSD capacity usage is critically overflowing, and the internal spares are running out. The cluster may soon become unavailable for writing. | | UsedSSDCapacityNoLongerOverflows | INFO | SSD capacity usage is no longer overflowing. | | UsedSSDCapacityOverflow | MAJOR | SSD capacity usage is overflowing, and the internal capacity spares are used. | ### Resources | **Type** | **Severity** | **Description** | | ------------------------------------------ | ------------ | ------------------------------------------------------------------ | | APIServerStarted | INFO | Successfully started the API server | | APIServerStartFailed | WARNING | Failed to start the API server | | BandwidthSelected | INFO | Bandwidth set for host | | CoreAllocated | INFO | Allocated core | | DeviceIsNotAValidNetworkDevice | WARNING | Device is not a valid network device | | DisabledNumaBalancing | INFO | Disabled NUMA Balancing | | DriverLoaded | INFO | Driver attached | | FailedToLoadDriver | WARNING | Failed to load the wekafs driver | | HangingHTTPRequest | MINOR | Hanging HTTP request detected. | | HttpServerFibersExhausted | MAJOR | Hanging HTTP requests exhausted all available fibers | | HugepagesAllocated | INFO | Hugepages allocated | | HugepagesAllocationRetries | WARNING | Hugepages allocation retried | | HugepagesAllocationStarted | INFO | Hugepages allocation started | | HugepagesAllocationTookTooLong | WARNING | Hugepages allocation takes an unexcepectedly long duration | | InactiveHostCannotJoinCluster | INFO | Inactive host cannot join the cluster | | LoadingStableResourcesFailed | INFO | Failed loading stable resources | | NetBufsExhausted | MAJOR | netbufs exhausted | | NetDevDriverReloadFailed | MINOR | Net device driver reload failed | | NetworkDeviceAllocated | INFO | Allocated network device | | NetworkDeviceHasNoIp | MAJOR | Network device not has no ip address | | NetworkDeviceNotUsedByAnySlots | MINOR | Network device not used by any slots | | NoIPsConfiguredForHostJoinWithNoDefaultNet | WARNING | No IP configured for the process (node) {nid} with no default-net. | | RevertToStableResources | INFO | Reverted to stable resources | | UnlimitedBandwidthSelected | INFO | Bandwidth set to unlimited | | WCGroupContainerEvent | MAJOR | Container Status | | WCGroupUsageMajorEvent | MAJOR | Container {resource} Status | | WCGroupUsageWarningEvent | WARNING | Container {resource} Status | ### S3 | **Type** | **Severity** | **Description** | | --------------------------------------- | ------------ | ------------------------------------------ | | S3AddBucketILMRuleEvent | INFO | S3 add bucket ILM rule | | S3AsssumeRoleEvent | INFO | S3 STS token created | | S3AttachIAMPolicyEvent | INFO | S3 attach IAM policy | | S3AuditWebhookDisabledEvent | INFO | The S3 Audit Webhook disabled | | S3AuditWebhookEnabledEvent | INFO | The S3 Audit Webhook enabled | | S3BucketDestroyedEvent | INFO | The S3 bucket destroyed | | S3BucketNotificationAdded | INFO | S3 bucket notification added | | S3BucketNotificationRemoved | INFO | S3 bucket notification removed | | S3ClusterCreated | INFO | An S3 cluster created | | S3ClusterDestroyed | INFO | The S3 cluster destroyed | | S3ClusterDestroyFailed | MAJOR | Failed to destroy the S3 cluster | | S3ClusterStatusActiveEvent | INFO | S3 Cluster is active | | S3ClusterStatusInactiveEvent | CRITICAL | S3 Cluster is inactive | | S3ClusterUpdated | INFO | The S3 cluster updated | | S3ContainerStateChangesEvent | INFO | S3 container status change | | S3ContainerStatusActiveEvent | INFO | S3 container active | | S3ContainerStatusInactiveEvent | MAJOR | S3 container inactive | | S3ContainerStatusOnlineEvent | INFO | S3 container online | | S3ContainerStatusSaturatedEvent | MAJOR | S3 container saturated | | S3CreateBucketEvent | INFO | The S3 bucket created | | S3CreateIAMPolicyEvent | INFO | The S3 created an IAM Policy | | S3CreateServiceAccountEvent | INFO | S3 create service account | | S3DestroyBucketEvent | INFO | The S3 bucket destroyed | | S3DetachIAMPolicyEvent | INFO | S3 detach IAM policy | | S3DrainEvent | INFO | S3 container drain | | S3KVAddedEvent | INFO | S3 Config Add Key | | S3KVRemovedEvent | INFO | S3 configuration remove key | | S3KVResetEvent | INFO | S3 KV store configuration reset | | S3MultipleContainersStatusInactiveEvent | CRITICAL | multiple S3 containers are inactive | | S3NotificationDropped | CRITICAL | S3 notifications dropped. | | S3NotificationTargetAdded | INFO | S3 notification-target added | | S3NotificationTargetCertificateAdded | INFO | S3 notification-target certificate added | | S3NotificationTargetCertificateRemoved | INFO | S3 notification-target certificate removed | | S3NotificationTargetRemoved | INFO | S3 notification-target removed | | S3NotificationTargetUpdated | INFO | S3 notification-target updated | | S3RemoveBucketILMRuleEvent | INFO | S3 remove bucket ILM rule | | S3RemoveIAMPolicyEvent | INFO | The S3 removed the IAM policy | | S3RemoveServiceAccountEvent | INFO | S3 remove service account | | S3ResetBucketILMRuleEvent | INFO | S3 reset bucket ILM rules | | S3SetBucketPolicyEvent | INFO | S3 set bucket policy | | S3UnDrainEvent | INFO | S3 container undrain | | SLBContainerStatusActiveEvent | INFO | SLB container active | | SLBContainerStatusInactiveEvent | MAJOR | SLB container inactive | ### Security | **Type** | **Severity** | **Description** | | ----------------------------------- | ------------ | ------------------------------------------------------------------------- | | BackendJoinTLSVerificationFailure | MINOR | A backend tried to join the cluster but failed the TLS verification check | | BackendNodeJoinSecurityPolicyDenied | WARNING | Backend join denied per security policy | | CaCertSet | INFO | The CA certificate was added to the cluster | | CaCertUnset | INFO | The CA certificate was unset | | ClientNodeJoinSecurityPolicyDenied | WARNING | Client join denied per security policy | | ContainerJoinSecretDenied | WARNING | The container failed to rejoin due to missing or incorrect join-secret | | FileSystemSecurityPoliciesChange | INFO | Filesystem security policies changed. | | JoinSecurityPoliciesUpdated | INFO | Join Security Policies for {mode} configuration change | | LocalTLSCertAdded | INFO | CertificateChange configuration change | | LocalTLSCertRemoved | INFO | CertificateChange configuration change | | LocalTLSCertUpdated | INFO | CertificateChange configuration change | | MountAccessDenied | WARNING | Mount Access Denied | | MountSecurityPolicyAllowed | INFO | Mount Access Allowed By Security Policy | | MountSecurityPolicyDenied | WARNING | Mount Access Denied By Security Policy | | OrgSecurityPoliciesChanged | INFO | The organization security policies have changed. | | OrgTokensRevoked | INFO | The organization API tokens have been revoked. | | SecurityPolicyAccessDenied | WARNING | Access denied per security policy | | SecurityPolicyCreated | INFO | Security Policy configuration change | | SecurityPolicyDeleted | INFO | Security Policy configuration change | | SecurityPolicyUpdated | INFO | Security Policy configuration change | | TLSSet | INFO | TLS was set | | TLSStrictnessUpdated | INFO | TLS strictness updated | | TLSUnset | INFO | TLS was unset | | TokenTimeoutsChange | INFO | Token timeouts for authentication were updated | ### Smb | **Type** | **Severity** | **Description** | | --------------------------------- | ------------ | ------------------------------------- | | SmbAdJoined | INFO | Active Directory configuration change | | SmbAdLeft | INFO | Active Directory configuration change | | SmbClusterConfigured | INFO | SMB cluster configuration change | | SmbClusterCreateCreated | INFO | SMB cluster configuration change | | SmbClusterDestroyed | INFO | SMB cluster configuration change | | SmbConfigGenerationUpdated | INFO | SMB Config configuration change | | SmbShareAdded | INFO | Share configuration change | | SmbShareConfigured | INFO | Share configuration change | | SmbShareHostnameACERemovedRemoved | INFO | SambaHostnameACE configuration change | | SmbShareHostnameACEResetDestroyed | INFO | SambaHostnameACE configuration change | | SmbShareRemoved | INFO | Share configuration change | | SmbTrustedDomainAdded | INFO | TrustedDomain configuration change | | SmbTrustedDomainRemoved | INFO | TrustedDomain configuration change | ### Statistics | **Type** | **Severity** | **Description** | | ----------------- | ------------ | ---------------------------------------- | | StatLimitExceeded | WARNING | A set limit on a statistics is exceeded. | ### System | **Type** | **Severity** | **Description** | | ---------------------------------- | ------------ | -------------------------------------------------------------------------------------------------------- | | BlockTaskAborted | INFO | The bucket task was aborted successfully. | | BlockTaskComplete | INFO | A bucket task completed successfully. | | BucketsCreated | INFO | The system has created buckets. | | ClusterTaskAborted | INFO | Cluster task aborted | | ClusterTaskAborting | INFO | Cluster task started aborting | | ClusterTaskPaused | INFO | Cluster task paused | | ClusterTaskResumed | INFO | Cluster task resumed | | ClusterTasksCpuLimitUpdated | INFO | Cluster tasks CPU limit set. | | ClusterwideTaskChanged | DEBUG | The cluster wide task has changed. | | DataServiceTaskFailedWithError | MINOR | Data service task failure | | DataServiceTaskWaitingForChildTask | INFO | Data service task waiting | | DsShardViewChanged | INFO | There is a change in the data service shard view | | GcHighExecutionTime | MINOR | GC execution time percent of total time is too high. | | GcTooManyScans | MINOR | Too many GC scans in a short period. | | HaveEnoughSSDCapacity | MINOR | Sufficient SSD capacity exists for all the provisioned filesystems. | | IOStarted | INFO | The system has started. | | IOStopped | INFO | The system has stopped. | | LeaderFailedToReachBaseline | CRITICAL | Reports from too many processes in the cluster failed to reach the newly elected cluster leader in time. | | NotEnoughSSDCapacity | CRITICAL | Need more SSD capacity for all the provisioned filesystems. | | QOSConfigReset | INFO | QoS configuration reset | | QOSConfigSet | INFO | QoS configuration set | | SleepyDeusExFiber | MAJOR | A deus-ex worker fiber is sleepy, which is a likely indication of a deadlock | | StartIORequested | INFO | The user has requested to start the IO. | | StopIORequested | INFO | The user has requested to stop the IO. | | SystemInfoReport | INFO | The management process (node) started, reporting OS info. | | ThreadPoolCanNotStartThread | MINOR | The reactor's thread pool failed to start a thread. | | TooManyFibers | MINOR | Too many fiber allocations. | ### Telemetry | **Type** | **Severity** | **Description** | | ---------------------------------- | ------------ | -------------------------------------- | | AuditTracesStatusChange | INFO | Audit traces status changed | | TelemetryExportAdded | INFO | Telemetry Export configuration change | | TelemetryExportDisabled | INFO | Telemetry Export Disabled | | TelemetryExportEnabled | INFO | Telemetry Export Enabled | | TelemetryExportRemoved | INFO | Telemetry Export configuration change | | TelemetryExportUpdated | INFO | Telemetry Export configuration change | | TelemetrySourcesAttachetToExport | INFO | Telemetry Sources Attached To Export | | TelemetrySourcesDetachedFromExport | INFO | Telemetry Sources Detached From Export | ### Traces | **Type** | **Severity** | **Description** | | ---------------------------------- | ------------ | -------------------------------------------- | | RemoteTracesDisabled | INFO | Remote traces disabled | | RemoteTracesEnabled | INFO | Remote traces enabled | | RemoteTraceStreamerEndpointInvalid | INFO | Invalid remote trace streamer endpoint | | TracesConfigurationActivated | INFO | Traces configuration change | | TracesConfigurationDeactivated | INFO | Traces configuration change | | TracesConfigurationReset | INFO | Traces configuration change | | TracesConfigurationUpdated | INFO | Traces configuration change | | TracesFreezeOnEventOfInterest | MAJOR | Traces have been frozen on event of interest | | TracesFreezePeriodReset | INFO | Traces freeze period has been reset | | TracesFreezePeriodSet | INFO | Traces freeze period has been set | | TracesVerbosityLevelChange | MAJOR | The Traces Verbosity level has been modified | ### Upgrade | **Type** | **Severity** | **Description** | | ---------------------------- | ------------ | -------------------------------------------------------- | | CleanupUpgradePhaseSkipped | MAJOR | Skipped Cleanup in Upgrade Phase: {currentPhase}. | | ClientUpgradeRequested | INFO | A client upgrade is requested. | | ComputeUpgradeFinished | INFO | The compute containers upgrade has finished. | | ComputeUpgradeInvoked | INFO | The compute containers upgrade has started. | | ComputeUpgradeStarted | INFO | The compute containers upgrade has started. | | ContainerUpgradeFailed | MAJOR | Container upgrade failed. | | ContainerUpgradeFinished | INFO | Container upgrade finished. | | ContainerUpgradeStarted | INFO | Container upgrade started. | | DataservUpgradeFinished | INFO | The dataserv containers upgrade has finished. | | DataservUpgradeStarted | INFO | The dataserv containers upgrade has started. | | DrivesUpgradeFinished | INFO | The drives containers upgrade has finished. | | DrivesUpgradeStarted | INFO | The drives containers upgrade has started. | | EventGeneratorTestWithReport | INFO | An event for the Event Generator Utility, with reporting | | ExternalUpgradeCancelled | INFO | The external upgrade has being cancelled. | | ExternalUpgradeFinished | INFO | The external upgrade has finished. | | ExternalUpgradeStarting | INFO | An external upgrade has started. | | FailureDomainUpgradeStarted | INFO | Started to upgrade containers of a failure domain. | | FinishedExternalHostUpgrade | INFO | The external server (host) upgrade has finished. | | FrontendUpgradeFinished | INFO | The frontend containers upgrade has finished. | | FrontendUpgradeStarted | INFO | The frontend containers upgrade has started. | | StartingExternalHostUpgrade | INFO | The external server (host) upgrade has started. | | TargetVersionChange | DEBUG | Target version has changed. | | UpgradePaused | INFO | Upgrade paused. | | UpgradeResumed | INFO | Upgrade resumed. | | UpgradeStatusChange | DEBUG | Upgrade status has changed. | | WekaVersionDowngraded | WARNING | The cluster is running a lower version. | ### User | **Type** | **Severity** | **Description** | | -------------------------------- | ------------ | -------------------------------------------- | | AccessGroupsDisabled | INFO | Access groups disabled | | AccessGroupsEnabled | INFO | Access groups enabled | | LDAPAuthDisabled | INFO | LDAP authentication disabled | | LDAPAuthEnabled | INFO | LDAP authentication enabled | | LDAPConfigUpdated | INFO | LDAP configuration updated | | UserCreated | INFO | The user is created. | | UserDeleted | INFO | The user is deleted. | | UserLoggedIn | INFO | User logged in | | UserLoginFailed | INFO | User login failed | | UserLoginLocked | MINOR | User login locked | | UserPasswordChanged | INFO | The user changed the password. | | UserPasswordChangedByAnotherUser | INFO | The administrator changed the user password. | | UserRoleChanged | INFO | The user role is changed. | --- # Agent Instructions: Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the `ask` query parameter: ``` GET https://docs.weka.io/4.4/operation-guide/events/events-list.md?ask= ``` The question should be specific, self-contained, and written in natural language. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.