List of statistics
Explore the various statistics the WEKA system produces, organized according to their respective categories.
Bucket
Type
Description
Units
BUCKET_START_TIME
Duration of bucket activation on step up
Startups
CHOKING_LEVEL_ALL
Throttling level applied on all types of IOs
%
CHOKING_LEVEL_NON_MUTATING
Throttling level applied on non-mutating only types of IOs
%
COALESCED_MAY_CREATE_EXTENT
Number of mayCreateExtent calls coalesced
Calls
DESTAGED_BLOCKS_COUNT
Number of destaged blocks per second
Blocks/Sec
DESTAGE_COUNT
Number of destages per second
Destages/Sec
DIR_MOVE_TIME
Time to complete a directory move
Ops
EXTENT_BLOCKS_COUNT
Difference in number of EXTENT blocks
Blocks
EXTENT_BLOCK_SEQUENCES
Histogram of the number of consecutive sequences of blocks in a single extent
Extents
FAIRNESS_DELAYED_MAY_CREATE_EXTENT
Number of mayCreateExtent calls not coalesced to prevent starvation
Calls
FREEABLE_LRU_BUFFERS
Number of unused blocks in LRU cache
Buffers
HASH_BLOCKS_COUNT
Difference in number of HASH blocks
Blocks
INODE_BLOCKS_COUNT
Difference in number of INODE blocks
Blocks
INTEGRITY_ISSUES
Number of filesystem integrity issues detected
Issues
JOURNAL_BLOCKS_COUNT
Difference in number of JOURNAL blocks
Blocks
ODH_COLLISIONS_ACCESS_CLOCK_STATES
Number of ODH items created with colliding hash in ACCESS_CLOCK_STATES ODH
Collisions
ODH_COLLISIONS_BIG_BLOB_MANIFEST
Number of ODH items created with colliding hash in BIG_BLOB_MANIFEST ODH
Collisions
ODH_COLLISIONS_DEFAULT_DIR_QUOTA
Number of ODH items created with colliding hash in DEFAULT_DIR_QUOTA ODH
Collisions
ODH_COLLISIONS_DIRECTORY
Number of ODH items created with colliding hash in DIRECTORY ODH
Collisions
ODH_COLLISIONS_DIR_QUOTA
Number of ODH items created with colliding hash in DIR_QUOTA ODH
Collisions
ODH_COLLISIONS_FLOCK_EXPIRED_FRONTENDS_WNID
Number of ODH items created with colliding hash in FLOCK_EXPIRED_FRONTENDS_WNID ODH
Collisions
ODH_COLLISIONS_FLOCK_EXPIRED_FRONTENDS
Number of ODH items created with colliding hash in FLOCK_EXPIRED_FRONTENDS ODH
Collisions
ODH_COLLISIONS_GRAVEYARD
Number of ODH items created with colliding hash in GRAVEYARD ODH
Collisions
ODH_COLLISIONS_INODES_PENDING_VALIDATIONS
Number of ODH items created with colliding hash in INODES_PENDING_VALIDATIONS ODH
Collisions
ODH_COLLISIONS_INODES_POTENTIAL_PENDING_DELETION
Number of ODH items created with colliding hash in INODES_POTENTIAL_PENDING_DELETION ODH
Collisions
ODH_COLLISIONS_MODIFY_CLOCK_STATES
Number of ODH items created with colliding hash in MODIFY_CLOCK_STATES ODH
Collisions
ODH_COLLISIONS_OBS_IMMEDIATE_RELEASE
Number of ODH items created with colliding hash in OBS_IMMEDIATE_RELEASE ODH
Collisions
ODH_COLLISIONS_OBS_RECLAMATION
Number of ODH items created with colliding hash in OBS_RECLAMATION ODH
Collisions
ODH_COLLISIONS_REFERENCE_RELOCATIONS
Number of ODH items created with colliding hash in REFERENCE_RELOCATIONS ODH
Collisions
ODH_COLLISIONS_SNAPSHOT_MEMBERS
Number of ODH items created with colliding hash in SNAPSHOT_MEMBERS ODH
Collisions
ODH_COLLISIONS_SNAP_LAYER_CAPACITY
Number of ODH items created with colliding hash in SNAP_LAYER_CAPACITY ODH
Collisions
ODH_COLLISIONS_SNAP_LAYER_SIZE_V4_2
Number of ODH items created with colliding hash in SNAP_LAYER_SIZE_V4_2 ODH
Collisions
ODH_COLLISIONS_SNAP_LAYER_SIZE_V4_3
Number of ODH items created with colliding hash in SNAP_LAYER_SIZE_V4_3 ODH
Collisions
ODH_COLLISIONS_STOW_DOWNLOAD_REDISTRIBUTE_PULL_STATE_V4_3
Number of ODH items created with colliding hash in STOW_DOWNLOAD_REDISTRIBUTE_PULL_STATE_V4_3 ODH
Collisions
ODH_COLLISIONS_STOW_DOWNLOAD_REDISTRIBUTE_V4_3
Number of ODH items created with colliding hash in STOW_DOWNLOAD_REDISTRIBUTE_V4_3 ODH
Collisions
ODH_COLLISIONS_STOW_UPLOAD_MANIFEST
Number of ODH items created with colliding hash in STOW_UPLOAD_MANIFEST ODH
Collisions
ODH_COLLISIONS_SV_CAPACITY_LEADER
Number of ODH items created with colliding hash in SV_CAPACITY_LEADER ODH
Collisions
ODH_COLLISIONS_UNLINKED_INODES
Number of ODH items created with colliding hash in UNLINKED_INODES ODH
Collisions
ODH_COLLISIONS
Number of ODH items created with colliding hash in all ODHs
Collisions
ODL_BLOCKS_COUNT
Difference in number of ODL blocks
Blocks
ODL_PAYLOAD_BLOCKS_COUNT
Difference in number of ODL_PAYLOAD blocks
Blocks
READS
Number of read operations per second
Ops/Sec
READ_BYTES
Number of bytes read per second
Bytes/Sec
READ_LATENCY
Average latency of READ operations
Microseconds
REGISTRY_COLLISIONS
Number of registry items created with colliding key
Collisions
REGISTRY_L1_BLOCKS_COUNT
Difference in number of REGISTRY_L1 blocks
Blocks
REGISTRY_L2_BLOCKS_COUNT
Difference in number of REGISTRY_L2 blocks
Blocks
REGISTRY_SEARCHES_COUNT
Number of registry searches per second
Queries/Sec
RESIDENT_BLOCKS_COUNT
Number of blocks in resident blocks table
Blocks
SINGLE_HOP_MISMATCH_RECOVERY
1HOP prefix mismatch recoveries
Issues
SINGLE_HOP_RDMA_MISMATCH_DPDK_FALLBACK
1HOP prefix mismatch RDMA fail
Issues
SNAPSHOT_CREATION_TIME
Time to complete a snapshot creation
Snapshots
SPATIAL_SQUELCH_BLOCKS_COUNT
Difference in number of SPATIAL_SQUELCH blocks
Blocks
SUCCESSFUL_DATA_WEDGINGS
Number of successful attempts to wedge data blocks in journal per second
Attempts/Sec
SUPERBLOCK_BLOCKS_COUNT
Difference in number of SUPERBLOCK blocks
Blocks
TEMPORAL_SQUELCH_BLOCKS_COUNT
Difference in number of TEMPORAL_SQUELCH blocks
Blocks
TRANSIENT_INTEGRITY_ISSUES
Number of transient filesystem integrity issues detected
Issues
UNSUCCESSFUL_DATA_WEDGINGS
Number of unsuccessful attempts to wedge data blocks in journal per second
Attempts/Sec
USER_DATA_BUFFERS_IN_USE
Number of data buffers used for serving ongoing IOs
Buffers
WRITES
Number of write operations per second
Ops/Sec
WRITE_BYTES
Number of byte writes per second
Bytes/Sec
WRITE_LATENCY
Average latency of WRITE operations
Microseconds
CPU
Type
Description
Units
CPU_UTILIZATION
The percentage of the CPU time used for handling I/Os
%
Charters
Type
Description
Units
DEDGRADED_TO_READER_RELINQUISHES
Charter relinquishes by reason
Charters
EAGER_RELINQUISHES
Charter relinquishes by reason
Charters
LRU_EXPIRED_RELINQUISHES
Charter relinquishes by reason
Charters
LRU_LENGTH_RELINQUISHES
Charter relinquishes by reason
Charters
OUT_OF_SPACE_RELINQUISHES
Charter relinquishes by reason
Charters
Choking
Type
Description
Units
CHOKING_LEVEL_ALL
Throttling level applied on all types of IOs, both mutating and non-mutating
Processes
CHOKING_LEVEL_NON_MUTATING
Throttling level applied on non-mutating only types of IOs
Processes
Clients
Type
Description
Units
CLIENTS_CONNECTED
Clients connected
Clients/Sec
CLIENTS_DISCONNECTED
The number of clients left or removed
Clients/Sec
CLIENTS_LEFT
The number of clients left
Clients/Sec
CLIENTS_RECONNECTED
The number of clients reconnected instead of their previous connection instance
Clients/Sec
CLIENTS_REMOVED
The number of clients removed
Clients/Sec
Config
Type
Description
Units
AVERAGE_CHANGES_IN_CHANGESET
The average number of changes in a changeset
Changes/Sec
AVERAGE_CHANGES_IN_GENERATION
The average number of changes in a generation
Changes/Sec
BACKEND_NODE_REJOIN_TIME
The number of backend rejoin attempts per completion time range
Number of rejoins
CHANGESET_COMMIT_LATENCY
The average latency of committing a configuration changeset
Microseconds
CLIENT_NODE_REJOIN_TIME
The number of clients rejoin attempts per completion time range
Number of rejoins
CONFIG_PROPAGATION_LATENCY
The latencies of propagation of a configuration generation
Generation
FetchLocalStateChangesCallType_INTERNAL_CONTINUE
Number of RPC calls to fetch local-state LRU changes, per overlay node type & call type
RPC Calls
FetchLocalStateChangesCallType_INTERNAL_RESTART_FROM_TAIL
Number of RPC calls to fetch local-state LRU changes, per overlay node type & call type
RPC Calls
FetchLocalStateChangesCallType_INTERNAL_RETRY_LAST_REQUEST
Number of RPC calls to fetch local-state LRU changes, per overlay node type & call type
RPC Calls
FetchLocalStateChangesCallType_LEAF_CONTINUE
Number of RPC calls to fetch local-state LRU changes, per overlay node type & call type
RPC Calls
FetchLocalStateChangesCallType_LEAF_RESTART_FROM_TAIL
Number of RPC calls to fetch local-state LRU changes, per overlay node type & call type
RPC Calls
FetchLocalStateChangesCallType_LEAF_RETRY_LAST_REQUEST
Number of RPC calls to fetch local-state LRU changes, per overlay node type & call type
RPC Calls
GENERATION_COMMIT_LATENCY
The average latency of committing a configuration generation to the RAFT log
Microseconds
HEARTBEAT_PROCESSING_TIME_OLD
The number of non-leader heartbeats per processing time range (OLD)
Number of heartbeats
HEARTBEAT_PROCESSING_TIME
The number of non-leader heartbeats per processing time range
Number of heartbeats
HISTOGRAM_LEADER_ITERATION_WAIT_DURATION_CONFIG_ALIGNMENT
Wait duration of leader iteration for all nodes to align on latest configuration generation
Leader iteration wait time
LEADER_HEARTBEAT_PROCESSING_TIME_OLD
The number of leader heartbeats per processing time range (OLD)
Number of heartbeats
LEADER_HEARTBEAT_PROCESSING_TIME
The number of leader heartbeats per processing time range
Number of heartbeats
LOCALSTATE_AGGREGATION_LATENCY
This period between clockSkewReportTime table's update by a management process and the time the leader sees it
Time is taken to aggregate LocalState in milliseconds
LOCAL_STATS_FETCH_GENERATION_LAGGING
The number of local-state generations that the parent fetch request still needs to read
local-state generations
OVERLAY_FULL_SHIFTS
The number of entire overlay shifts
Changes
OVERLAY_INCREMENTAL_SHIFTS
The number of incremental overlay shifts
Changes
OVERLAY_TRACKER_INCREMENTALS
The number of incremental OverlayTracker applications
Changes
OVERLAY_TRACKER_RESYNCS
The number of OverlayTracker full-resyncs
Changes
TOTAL_CHANGESETS_COMMITTED
The total number of committed changesets
Change Sets
TOTAL_COMMITTED_CHANGES
The total number of committed configuration change sets
Changes
TOTAL_CONFIG_SNAPSHOT_PULLS
The total number of config snapshot pulls
Pulls
TOTAL_GENERATIONS_COMMITTED
The number of committed generations
Generations
Dataservice
Type
Description
Units
DIFFLIST_GET_LATENCY
Average latency of getDifflist
Microseconds
DIFFLIST_GET_MANIFEST_LATENCY
Average latency of getDifflist getManifest
Microseconds
DIFFLIST_GET_MANIFEST_OPS
Number of getDifflist getManifest
Ops/Sec
DIFFLIST_GET_MANIFEST_PER_GETLIST_LATENCY
Average latency of getDifflist getmanifest per getdifflist
Microseconds
DIFFLIST_GET_MANIFEST_PER_GETLIST_OPS
Number of getDifflist getmanifest per getdifflist
Ops/Sec
DIFFLIST_GET_OPS
Number of getDifflist
Ops/Sec
DIFFLIST_RESOLVEPATH_LATENCY
Average latency of getDifflist resolvepath
Microseconds
DIFFLIST_RESOLVEPATH_OPS
Number of getDifflist resolvepath
Ops/Sec
DIFFLIST_RESOLVE_PATH_BATCH_LATENCY
Average latency of getDifflist resolve-path per batch
Microseconds
DIFFLIST_RESOLVE_PATH_BATCH_OPS
Number of getDifflist resolve-path per bacth
Ops/Sec
QUOTAS_MARKED
Number of directory quotas marked
Quotas
QUOTA_TASK_ADD_DIR_ENTRIES
Number of entries added for directory quota task
Ops
QUOTA_TASK_CREATES
Number of directory quota tasks created
Ops
QUOTA_TASK_DELETES
Number of directory quota tasks removed
Ops
QUOTA_TASK_DELETE_DIR_ENTRIES
Number of entries removed for directory quota task
Ops
QUOTA_TASK_FAILED_STAMPS
Number of failed quota coloring stamp operations
Ops
QUOTA_TASK_FIBERS
Number of directory quota task fibers spawned per second
Fibers
QUOTA_TASK_READDIR_LATENCY
Average latency of directory quota task readdir operations
Microseconds
QUOTA_TASK_READDIR_OPS
Number of directory quota task readdir operations per second
Ops/Sec
QUOTA_TASK_RUNTIME
Average runtime of directory quota task fibers
Microseconds
QUOTA_TASK_STAMPS
Number of directory quota stamp operations per second
Ops/Sec
QUOTA_TASK_STAMP_LATENCY
Average latency of directory quota task stamp operations
Microseconds
QUOTA_TASK_SUCCESSFUL_STAMPS
Number of successful directory quota stamp operations
Ops
Frontend
Type
Description
Units
FE_IDLE_CYCLES
The number of idle cycles on the frontend
Cycles/Sec
FE_IDLE_TIME
The percentage of the CPU time not used for handling I/Os on the frontend
%
JRPC
Type
Description
Units
JRPC_SERVER_CALLS_CLIENT_DOES_NOT_SUPPORT_QOS
The number of JRPC calls made from a client that does not support JRPC QoS
Requests/Sec
JRPC_SERVER_CALLS_CLIENT_SUPPORTS_QOS
The number of JRPC calls made from a client that supports JRPC QoS
Requests/Sec
JRPC_SERVER_CALLS_QOS_DECLINED
The number of JRPC calls where server returns TOO_MANY_REQUESTS (QoS declined to run a method)
Requests/Sec
JRPC_SERVER_PROCESSING_AVG
The average time the JRPC server processed the JRPC requests.
Microseconds
JRPC_SERVER_PROCESSING_TIME
The number of JRPC requests processed by the server per each time range.
Requests
Memory
Type
Description
Units
GC_CURRENT
The process (node) GC memory size, current in sample time.
Bytes
GC_PEAK
The process (node) GC memory size, peak over 1-minute intervals.
Bytes
RSS_CURRENT
The process (node) memory resident size, current in sample time.
MB
RSS_PEAK
The process (node) memory resident size, peak over process lifetime.
MB
NODE_TRANSITIONS
Type
Description
Units
JOINING_FENCED_REASON_COUNTS
Counts of reasons JOINING nodes were fenced
Occurrences/Sec
JOINING_TO_UP_TRANSITIONS
Number of nodes transitioned from JOINING to UP.
Nodes
SYNCING_TO_JOINING_TRANSITIONS
Number of nodes transitioned from SYNCING to JOINING.
Nodes
SYNC_TO_JOIN_FAILURE_COUNTS
Counts of SYNCING to JOINING failures categorized by reason
Occurrences/Sec
UP_FENCED_REASON_COUNTS
Counts of reasons UP nodes were fenced
Occurrences/Sec
Network
Type
Description
Units
ACKS_LOST
Number of lost ACK packets
Packets/Sec
ACKS_REORDERED
Number of reordered ACK packets
Packets/Sec
BAD_RECV_CSUM
Number of packets received with a bad checksum
Packets/Sec
CORRUPT_PACKETS
Number of packets received and deemed corrupted
Packets/Sec
DOUBLY_RECEIVED_PACKETS
Number of packets that were received multiple times
Packets/Sec
DROPPED_LARGE_PACKETS
Number of large packets dropped in the socket backend
Packets/Sec
DROPPED_PACKETS
Number of packets received that we dropped
Packets/Sec
ECN_ENCOUNTERED
Number of ECN Encountered packets
Packets/Sec
FAULT_RECV_DELAYED_PACKETS
Number of received packets delayed due to a fault injection
Packets/Sec
FAULT_RECV_DROPPED_PACKETS
Number of received packets dropped due to a fault injection
Packets/Sec
FAULT_SENT_DELAYED_PACKETS
Number of sent packets delayed due to a fault injection
Packets/Sec
FAULT_SENT_DROPPED_PACKETS
Number of sent packets dropped due to a fault injection
Packets/Sec
FRAGMENTATION_DUPS
Number of packets duplicated during fragmentation
Packets/Sec
GOODPUT_RX_RATIO
Percentage of goodput RX packets out of total data packets received
%
GOODPUT_TX_RATIO
Percentage of goodput TX packets out of total data packets sent
%
GW_MAC_RESOLVE_FAILURES
Number of times we failed to ARP resolve the gateway IP
Failures
GW_MAC_RESOLVE_SUCCESSES
Number of times we succeeded in ARP resolve the gateway IP
Successes
INVALID_FIRST_FRAGMENT
Number of times we got an invalid first fragment
Packets/Sec
MBUF_DUP_COUNT
Numer of Duplicate mbufs found
Occurrences
MBUF_DUP_ITER
Duplicate mbuf check completions
Occurrences
NDP_DAD_RECV_ADDR_CONFLICTS
NDP DAD Receive Address Conflict Detected
Packets/Sec
NDP_DAD_RECV_NO_CONFLICTS
NDP DAD Receive No Conflict
Packets/Sec
NODE_RECONNECTED
Number of reconnections
Reconnects/Sec
PACKETS_FAILING_COMBINE
Number of packets received that failed buffer combining
Packets/Sec
PACKETS_NEEDING_ALIGN
Number of packets received that needed alignment adjustment
Packets/Sec
PACKETS_NEEDING_COMBINE
Number of packets received that needed buffer combining
Packets/Sec
PACKETS_PUMPED
Number of packets received in each call to recvPackets
Batches
PACKETS_VLAN_INSERTED_HW
Number of packets sent with hardware inserted VLAN tag
Packets/Sec
PACKETS_VLAN_INSERTED_SW
Number of packets sent with software inserted VLAN tag
Packets/Sec
PACKETS_VLAN_STRIPPED_HW
Number of packets received with hardware stripped VLAN tag
Packets/Sec
PACKETS_VLAN_STRIPPED_SW
Number of packets received with software stripped VLAN tag
Packets/Sec
PACKET_ALIGN_BYTES_COPIED
Number of bytes copied during receive for packet alignment
Bytes/Sec
PACKET_COMBINE_BYTES_COPIED
Number of bytes copied during receive packet buffer combining
Bytes/Sec
PEER_RTT_BACKEND
RTT histogram
Microseconds
PEER_RTT_CLIENT
RTT histogram
Microseconds
POISON_DETECTED_EXPECTED
Expected number of poisoned netbufs detected
Occurrences
POISON_DETECTED_UNEXPECTED
Unexpected number of poisoned netbufs detected
Occurrences
POISON_DETECTED
Number of poisoned netbufs detected
Occurrences
PORT_EXT_RX_PACKETS
Number of external packets received
Packets/Sec
PORT_RX_BYTES
Number of bytes received
Bytes/Sec
PORT_RX_ERRORS
Number of packet RX errors
Packets/Sec
PORT_RX_MISSED
Number of packets lost due to RX queue full
Packets/Sec
PORT_RX_NO_MBUFS
Number of packets lost due to no mbufs
Packets/Sec
PORT_RX_PACKETS
Number of packets received
Packets/Sec
PORT_TX_BYTES
Number of bytes transmitted
Bytes/Sec
PORT_TX_ERRORS
Number of packet TX errors
Packets/Sec
PORT_TX_PACKETS
Number of packets transmitted
Packets/Sec
PUMPS_TXQ_FULL
Number of times we couldn't send any new packets to the NIC queue
Pumps/Sec
PUMPS_TXQ_PARTIAL
Number of times we only sent some of our queued packets to the NIC queue
Pumps/Sec
PUMP_DURATION
Duration of each pump
Requests
PUMP_INTERVAL
Interval between pumps
Requests
RDMA_ADD_CHUNK_FAILURES
Number of RDMA cookie setting failurs
Failures/Sec
RDMA_AHCACHE_POPULATIONS
Number of RDMA RDMA AH cache population attempts
Attempts/Sec
RDMA_BINDING_FAILOVERS
Number of RDMA High-Availability fail-overs
Fail-overs/Sec
RDMA_CANCELED_COMPLETIONS
Number of RDMA completions that were canceled
Completions/Sec
RDMA_CLIENT_BINDING_INVALIDATIONS
Number of RDMA client binding invalidations
Invalidations/Sec
RDMA_COMPLETIONS
Number of RDMA requests that were completed
Completions/Sec
RDMA_COMP_DURATION
Histogram of RDMA completion duration times
Requests
RDMA_COMP_FAILURES
Number of RDMA requests that were completed with an error
Failures/Sec
RDMA_COMP_LATENCY
Average time of RDMA requests completion
Microseconds
RDMA_COMP_STATUSES
Histogram of RDMA completion statuses
Completions/Sec
RDMA_FAILED_AHCACHE_POPULATIONS
Number of failed RDMA AH cache population attempts
Failed Attempts/Sec
RDMA_FALLBACK_WHILE_AH_POPULATE
Number of fallbacks from RDMA due to AH cache population in progress
Fallbacks/Sec
RDMA_NET_ERR_RETRY_EXCEEDED
Number of RDMA requests with error retries exceeded
Occurrences/Sec
RDMA_POOL_ALLOC_FAILED
Number of times an RDMA request was not issued due to a pool allocation failure
Failures/Sec
RDMA_POOL_LOW_CAPACITY
Number of times an RDMA request was not issued due to low RDAM pool memory
Failures/Sec
RDMA_POOL_MBUF_LEAKED
RDMA leaked mbufs
Occurrences
RDMA_PORT_WAITING_FIBERS
Number of fibers pending to send an RDMA request
Waiting fibers
RDMA_REQUESTS
Number of RDMA requests sent to the NIC
Requests/Sec
RDMA_RX_BYTES
Number of bytes received with RDMA
Bytes/Sec
RDMA_SERVER_BINDING_RESTARTS
Number of RDMA server binding restarts
Restarts/Sec
RDMA_SERVER_FAILED_BINDING_RESTARTS
Number of failed RDMA server binding restarts
Failed Restarts/Sec
RDMA_SERVER_RECV_FAILURES
Number of failed RDMA server-side receive attempts
Failures/Sec
RDMA_SERVER_SEND_FAILURES
Number of failed RDMA server-side send attempts
Failures/Sec
RDMA_SUBMIT_FAILURES
Number of RDMA submit failures, likely indicating a fabric issue
Failures/Sec
RDMA_SUBMIT_TIMEOUTS
Number of RDMA submit timeouts
Timeouts/Sec
RDMA_TX_BYTES
Number of bytes sent with RDMA
Bytes/Sec
RDMA_WAIT_INTERRUPTED
RDMA Wait interruptions
Issues
RDMA_WAIT_PREMATURE_WAKEUP
RDMA Wait for premature wakeup
Issues
RDMA_WAIT_TIMEOUT
RDMA Wait timeouts
Issues/Sec
RECEIVED_ACK_PACKETS
Number of received ack packets
Packets/Sec
RECEIVED_CONTROL_PACKETS
Number of received control packets
Packets/Sec
RECEIVED_DATA_PACKETS
Number of received data packets
Packets/Sec
RECEIVED_PACKETS
Number of packets received
Packets/Sec
RECEIVED_PACKET_GENERATIONS
The generation ("resend count") of the first incarnation of the packet seen by the receiver (indicates packet loss)
Packets
RECEIVED_PING_PACKETS
Number of received ping packets
Packets/Sec
RECEIVED_PONG_PACKETS
Number of received pong packets
Packets/Sec
RECEIVED_REJECT_PACKETS
Number of received reject packets
Packets/Sec
RECEIVED_SYNC_PACKETS
Number of received sync packets
Packets/Sec
REORDERED_PACKETS
Number of reordered packets
Packets/Sec
RESEND_BATCH_SIZE
Number of packets sent in a resend batch
Batches
RESENT_DATA_PACKETS
Number of data packets resent
Packets/Sec
SEND_BATCH_SIZE_BYTES
Number of bytes sent in a first send batch
Batches
SEND_BATCH_SIZE
Number of packets sent in a first send batch
Batches
SEND_QUEUE_TIMEOUTS
Number of packets canceled due to envelope timeout and were not in the send window
Packets/Sec
SEND_WINDOW_TIMEOUTS
Number of packets canceled due to envelope timeout while in the send window
Packets/Sec
SENT_ACKS
Number of ACK packets sent
Packets/Sec
SENT_CONTROL_PACKETS
Number of control packets sent
Packets/Sec
SENT_DATA_PACKETS
Number of data packets sent
Packets/Sec
SENT_PACKETS
Number of sent packets
Packets/Sec
SENT_REJECTS
Number of rejects sent
Packets/Sec
SHORT_CIRCUIT_SENDS
Number of packets sent to the same node
Packets/Sec
SLOW_PATH_CSUM
Number of packets that went through checksum calculation on the CPU
Packets/Sec
TIMELY_RESENDS
Number of packets resent due to timely resend
Packets/Sec
TIME_TO_ACK
Histogram of time to acknowledge a data packet
Requests
TIME_TO_FIRST_SEND
Time from queueing to first send
Requests
UCX_RXQ_FULL
UCX Drop RXQ Full
Packets/Sec
UCX_SEND_CB
UCX Send Callback
Packets/Sec
UCX_SEND_ERROR
UCX Send Error
Packets/Sec
UCX_SENT_PACKETS_ASYNC
UCX Sent Asynchronously
Packets/Sec
UCX_SENT_PACKETS_IMMEDIATE
UCX Sent Immediately
Packets/Sec
UCX_TXQ_FULL
UCX Drop TXQ Full
Packets/Sec
UDP_SENDMSG_FAILED_EAGAIN
Number of packets that failed to be sent on the socket backend with EAGAIN
Packets/Sec
UDP_SENDMSG_FAILED_OTHER
Number of packets that failed to be sent on the socket backend with an unknown error
Packets/Sec
UDP_SENDMSG_PARTIAL_SEND
Number of packets that we failed to send, but in the same pump, some packets were sent
Packets/Sec
UNACKED_RESENDS
Number of packets resent after receiving an ack
Packets/Sec
ZERO_CSUM
Number of checksum zero received
Packets/Sec
Operations
Type
Description
Units
ACCESS_LATENCY
Average latency of ACCESS operations
Microseconds
ACCESS_OPS
Number of ACCESS operations per second
Ops/Sec
COMMIT_LATENCY
Average latency of COMMIT operations
Microseconds
COMMIT_OPS
Number of COMMIT operations per second
Ops/Sec
CREATE_LATENCY
Average latency of CREATE operations
Microseconds
CREATE_OPS
Number of CREATE operations per second
Ops/Sec
FILEATOMICOPEN_LATENCY
Average latency of FILEATOMICOPEN operations
Microseconds
FILEATOMICOPEN_OPS
Number of FILEATOMICOPEN operations per second
Ops/Sec
FILECLOSE_LATENCY
Average latency of FILECLOSE operations
Microseconds
FILECLOSE_OPS
Number of FILECLOSE operations per second
Ops/Sec
FILEOPEN_LATENCY
Average latency of FILEOPEN operations
Microseconds
FILEOPEN_OPS
Number of FILEOPEN operations per second
Ops/Sec
FLOCK_LATENCY
Average latency of FLOCK operations
Microseconds
FLOCK_OPS
Number of FLOCK operations per second
Ops/Sec
FSINFO_LATENCY
Average latency of FSINFO operations
Microseconds
FSINFO_OPS
Number of FSINFO operations per second
Ops/Sec
GETATTR_LATENCY
Average latency of GETATTR operations
Microseconds
GETATTR_OPS
Number of GETATTR operations per second
Ops/Sec
LINK_LATENCY
Average latency of LINK operations
Microseconds
LINK_OPS
Number of LINK operations per second
Ops/Sec
LOOKUP_LATENCY
Average latency of LOOKUP operations
Microseconds
LOOKUP_OPS
Number of LOOKUP operations per second
Ops/Sec
MKDIR_LATENCY
Average latency of MKDIR operations
Microseconds
MKDIR_OPS
Number of MKDIR operations per second
Ops/Sec
MKNOD_LATENCY
Average latency of MKNOD operations
Microseconds
MKNOD_OPS
Number of MKNOD operations per second
Ops/Sec
OPS
Total number of operations
Ops/Sec
PATHCONF_LATENCY
Average latency of PATHCONF operations
Microseconds
PATHCONF_OPS
Number of PATHCONF operations per second
Ops/Sec
READDIR_LATENCY
Average latency of READDIR operations
Microseconds
READDIR_OPS
Number of READDIR operations per second
Ops/Sec
READLINK_LATENCY
Average latency of READLINK operations
Microseconds
READLINK_OPS
Number of READLINK operations per second
Ops/Sec
READS
Number of read operations per second
Ops/Sec
READ_BYTES
Number of bytes read per second
Bytes/Sec
READ_DURATION
The number of reads per completion duration
Reads
READ_LATENCY
Average latency of READ operations
Microseconds
REMOVE_LATENCY
Average latency of REMOVE operations
Microseconds
REMOVE_OPS
Number of REMOVE operations per second
Ops/Sec
RENAME_LATENCY
Average latency of RENAME operations
Microseconds
RENAME_OPS
Number of RENAME operations per second
Ops/Sec
RMDIR_LATENCY
Average latency of RMDIR operations
Microseconds
RMDIR_OPS
Number of RMDIR operations per second
Ops/Sec
SETATTR_LATENCY
Average latency of SETATTR operations
Microseconds
SETATTR_OPS
Number of SETATTR operations per second
Ops/Sec
STATFS_LATENCY
Average latency of STATFS operations
Microseconds
STATFS_OPS
Number of STATFS operations per second
Ops/Sec
SYMLINK_LATENCY
Average latency of SYMLINK operations
Microseconds
SYMLINK_OPS
Number of SYMLINK operations per second
Ops/Sec
THROUGHPUT
Number of byte read/writes per second
Bytes/Sec
UNLINK_LATENCY
Average latency of UNLINK operations
Microseconds
UNLINK_OPS
Number of UNLINK operations per second
Ops/Sec
WRITES
Number of write operations per second
Ops/Sec
WRITE_BYTES
Number of byte writes per second
Bytes/Sec
WRITE_DURATION
The number of writes per completion duration
Writes
WRITE_LATENCY
Average latency of WRITE operations
Microseconds
Platform
Type
Description
Units
REPORTED_MMAP_ALLOCS
Number of reported mmap alocattions
Allocations
REPORTED_MMAP_ALLOC_MEM
Memory allocated through reportedMmaps, in bytes.
Bytes
REPORTED_MMAP_DEALLOCS
Number of reported mmap dealocattions
Deallocations
REPORTED_MMAP_RESERVED_MEM
Memory reserved for reportedMmaps, in bytes.
Bytes
Processes
Type
Description
Units
ABRUPT_EXITS
How many abrupt exits of a process (node) occurred
Abrupt process exits
PEER_CONFIGURE_FAILURES
How many times the node failed to configure peers to sync with them
Peer configure failures
RAFT
Type
Description
Units
Bucket_LEADER_CHANGES
Changes of leader
Changes
Bucket_REQUESTS_COMPLETED
Requests to leader completed successfully
Requests
Configuration_LEADER_CHANGES
Changes of leader
Changes
Configuration_REQUESTS_COMPLETED
Requests to leader completed successfully
Requests
Invalid_LEADER_CHANGES
Changes of leader
Changes
Invalid_REQUESTS_COMPLETED
Requests to leader completed successfully
Requests
RAFT_BYTES_WRITTEN
Number of writes written to disk for RAFT
Bytes
SYNCLOG_TIMEOUTS
The number of timeouts of syncing logs to a process
Timeouts
Test_LEADER_CHANGES
Changes of leader
Changes
Test_REQUESTS_COMPLETED
Requests to leader completed successfully
Requests
RAID
Type
Description
Units
IS_BLOCK_USED_FREE_LATENCY
Average latency of handling an isBlockUsed of a free block
Micros
IS_BLOCK_USED_FREE
Number of isBlockUsed returning free
Blocks/Sec
IS_BLOCK_USED_USED_LATENCY
Average latency of handling an isBlockUsed of a used block
Micros
IS_BLOCK_USED_USED
Number of isBlockUsed returning used
Blocks/Sec
NVKV_RECOVERY_NETBUF_REREAD_UNEQUAL
Number of unequal netbufs encountered that caused NVKV recovery to restart
Blocks/Sec
RAID_ALLOCATION_FAILED_HOLES
Slots failed to be allocated and were left as holes
Holes/Sec
RAID_BLOCKS_IN_PREPARED_STRIPE
Free blocks in prepared stripe
Blocks
RAID_CHUNKS_CLEANED_BY_SHIFT
Dirty chunks cleaned by being shifted out
Occurences
RAID_CHUNKS_SHIFTED
Dirty chunks that shifted out
Occurences
RAID_COMMITTED_STRIPES
Number of stripes written
Stripes
RAID_COMPRESSED_BLOCKS_WRITTEN
Physical blocks are written containing compressed data
Blocks/Sec
RAID_CORRUPTION_RECOVERY_FAILURE
Corrupt data could not be recovered
Occurences
RAID_PLACEMENT_ALLOC_PlacementSpace0
Number of placement allocations
Occurrences/Sec
RAID_PLACEMENT_ALLOC_PlacementSpaceN_Compressed
Number of placement allocations
Occurrences/Sec
RAID_PLACEMENT_ALLOC_PlacementSpaceN_Uncompressed
Number of placement allocations
Occurrences/Sec
RAID_PLACEMENT_RETIRE_PlacementSpace0
Number of placement retirements
Occurrences/Sec
RAID_PLACEMENT_RETIRE_PlacementSpaceN_Compressed
Number of placement retirements
Occurrences/Sec
RAID_PLACEMENT_RETIRE_PlacementSpaceN_Uncompressed
Number of placement retirements
Occurrences/Sec
RAID_PLACEMENT_SWITCHES
Number of placement switches
Switches
RAID_READ_BATCHES_PER_REQUEST_HISTOGRAM
Histogram of the number of batches of stripes read in a single request
Request
RAID_READ_BLOCKS_STRIPE_HISTOGRAM
Histogram of the number of blocks read from a single stripe
Reads
RAID_READ_BLOCKS
Number of blocks read by the RAID
Blocks/Sec
RAID_READ_DEGRADED
Degraded mode reads
Blocks/Sec
RAID_READ_FREE
Read Free
Occurences
RAID_READ_IOS
Raw read blocks performed by the RAID
Blocks/Sec
RAID_STALE_WRITES_DETECTED
Stale write detected in read
Occurences
RAID_STALE_WRITES_REPROTECTIONS
Stale write reprotections in read
Occurences
WRONG_DRIVE_DELTAS
Delta segments are written to the wrong drive
Blocks/Sec
WRONG_DRIVE_REFS
Reference segments are written to the wrong drive
Blocks/Sec
RPC
Type
Description
Units
CLIENT_CANCELED_REQUESTS
Number of requests canceled by the client
Calls/Sec
CLIENT_DROPPED_RESPONSES
Number of responses dropped by the client
Calls/Sec
CLIENT_ENCRYPTION_AUTH_FAILURES
Number of authentication failures by the client
Calls/Sec
CLIENT_MISSING_ENCRYPTION_KEY
Number of times the client was missing an encryption key
Calls/Sec
CLIENT_RECEIVED_EXCEPTIONS
Number of exceptions received by the client
Calls/Sec
CLIENT_RECEIVED_RESPONSES
Number of responses received by the client
Calls/Sec
CLIENT_RECEIVED_TIMEOUTS
Number of timeouts experienced by the client
Calls/Sec
CLIENT_ROUNDTRIP_AVG_LOW
Roundtrip average of client low-priority RPC calls
Microseconds
CLIENT_ROUNDTRIP_AVG_NORM
Roundtrip average of client normal priority RPC calls
Microseconds
CLIENT_ROUNDTRIP_AVG
Roundtrip average of client normal and low priority RPC calls
Microseconds
CLIENT_RPC_CALLS_DOWNGRADED
Number of client-downgraded RPC calls
RPC/Sec
CLIENT_RPC_CALLS_LOW
Number of low-priority RPC calls
RPC/Sec
CLIENT_RPC_CALLS_NORM
Number of normal priority RPC calls
RPC/Sec
CLIENT_RPC_CALLS
Number of all priorities of RPC calls
RPC/Sec
CLIENT_SENT_REQUESTS
Number of requests sent by the client
Calls/Sec
DEUS_EX_MBUF_LIMITED
Number of RPCs slow down due to low MBuf reserves
Ops/Sec
DEUS_EX_NOT_EMPTY
Number of RPCs put in DeusEx to preserve RPC order
Ops/Sec
DEUS_EX_NO_FIBERS
Number of RPCs put in DeusEx due to lack of global fibers
Ops/Sec
DEUS_EX_RPC_MAX_FIBERS
Number of RPCs put in DeusEx due to RPC max fibers
Ops/Sec
FIRST_RESULTS
Number of first results per second
Ops/Sec
MBUF_LIMITED_SLEEP
Number of times wait due to low MBuf reserves
Actions/Sec
RPC_ENCRYPTION_SETUP_FAILURES
Number of encryption key setup failures
Failures
SERVER_ABORTS
Number of server received aborts
Calls/Sec
SERVER_DROPPED_REQUESTS
Number of requests dropped by the server
Calls/Sec
SERVER_ENCRYPTION_AUTH_FAILURES
Number of encryption authentication failures at the server
Calls/Sec
SERVER_MISSING_ENCRYPTION_KEY
Number of requests missing encryption key at the server
Calls/Sec
SERVER_PROCESSING_AVG
Average time to process server RPC calls
Microseconds
SERVER_PROCESSING_TIME
Histogram of the time it took the server to process a request
RPCs
SERVER_REJECTS
Number of times the server rejected a request
Calls/Sec
SERVER_RPC_CALLS_UPGRADED
Number of server-upgraded RPC calls
RPC/Sec
SERVER_RPC_CALLS
Number of server RPC calls
RPC/Sec
SERVER_SENT_EXCEPTIONS
Number of exceptions sent by the server as a response
Calls/Sec
SERVER_SENT_RESPONSES
Number of responses the server sent
Calls/Sec
SERVER_UNENCRYPTED_REFUSALS
Number of requests refused due to missing encryption at the server
Calls/Sec
TIME_TO_FIRST_RESULT
Average latency to the first result of a MultiCall
Microseconds
Reactor
Type
Description
Units
AVG_QUEUE_TIME_FLEX_TASKS
Average queue time of deferred flex tasks
Cycles
BACKGROUND_CYCLES
Number of cycles spent in background fibers
Cycles/Sec
BACKGROUND_FIBERS
Number of background fibers that are ready to run and eager to get CPU cycles
Fibers
BACKGROUND_TIME
The percentage of the CPU time used for background operations
%
BucketInvocationState_CAPACITY
Number of data structures allocated to the BucketInvocationState pool
Structs
BucketInvocationState_STRUCT_SIZE
Number of bytes in each struct of the BucketInvocationState pool
Bytes
BucketInvocationState_USED
Number of structs in the BucketInvocationState pool that are currently being used
Structs
CPU_HANGS_AND_KNOWN_HOGGER
Number of CPU hangs detected while known hogger
Hangs/Sec
CPU_HANGS
Number of CPU hangs detected
Hangs/Sec
CYCLES_PER_SECOND
Number of cycles the CPU runs per second
Cycles/Sec
DEFUNCT_FIBERS
Number of defunct buffers, which are just memory structures allocated for future fiber needs
Fibers
DeferredTask2_CAPACITY
Number of data structures allocated to the DeferredTask2 pool
Structs
DeferredTask2_STRUCT_SIZE
Number of bytes in each struct of the DeferredTask2 pool
Bytes
DeferredTask2_USED
Number of structs in the DeferredTask2 pool that are currently being used
Structs
EXCEPTIONS
Number of exceptions caught by the reactor
Exceptions/Sec
FLEX_TASKS_INLINE_FULLQUEUE
Number of flex tasks run inline on main thread due to full defer queue
Invocations/Sec
FLEX_TASKS_INLINE_LIGHTLOAD
Number of flex tasks run inline on main thread due to light load
Invocations/Sec
FLEX_TASKS_INLINE_OVERRIDE
Number of flex tasks run inline on main thread due to override
Invocations/Sec
FLEX_TASKS_LATENCY
Histogram of flex tasks latency
usecs
FLEX_TASKS_QUEUED_RUN_MAINTHREAD
Number of deferred flex tasks run by main thread
Invocations/Sec
FLEX_TASKS_QUEUED_RUN_THREADPOOL
Number of deferred flex tasks run on thread pool
Invocations/Sec
FLEX_TASKS_QUEUED
Number of deferred flex tasks queued
Invocations/Sec
FLEX_TASKS_SUBMITTED
Number of flex tasks submitted
Invocations/Sec
IDLE_CALLBACK_INVOCATIONS
Number of background work invocations
Invocations/Sec
IDLE_CYCLES
Number of cycles spent in idle
Cycles/Sec
IDLE_TIME
The percentage of the CPU time not used for handling I/Os
%
LINGERING_FIBERS
Number of LINGERING fibers
Fibers
MAIN_THREAD_DEFERRED_FLEX_TASK_CYCLES
Number of cycles main thread spent running deferred flex tasks
Cycles/Sec
MAIN_THREAD_INLINE_FLEX_TASKS_AVG_RUNTIME
Average runtime of inline flex tasks run by the main thread
Cycles
MAIN_THREAD_INLINE_FLEX_TASK_CYCLES
Number of cycles main thread spent running inline flex tasks
Cycles/Sec
MAIN_THREAD_QUEUED_FLEX_TASKS_AVG_RUNTIME
Average runtime of deferred flex tasks run by the main thread
Cycles
NODE_CONTEXT_SWITCHES
Number of context switches.
Switches
NODE_HANG
The number of process (node) hangs per hang time range.
Number of hangs
NODE_POLL_TIME
Time of scheduler stats polling.
usecs
NODE_RUN_PERCENTAGE
Percentage of time process is running
percentage
NODE_RUN_TIME
Time process is running.
usecs
NODE_WAIT_PERCENTAGE
Percentage of time process is waiting on waitqueue
percentage
NODE_WAIT_TIME
The Time process is waiting on the wait queue.
usecs
OUTRAGEOUS_HOGGERS
Number of hoggers taking an excessive amount of time to run
Invocations
ObsBucketManagement_CAPACITY
Number of data structures allocated to the ObsBucketManagement pool
Structs
ObsBucketManagement_STRUCT_SIZE
Number of bytes in each struct of the ObsBucketManagement pool
Bytes
ObsBucketManagement_USED
Number of structs in the ObsBucketManagement pool that are currently being used
Structs
ObsGateway_CAPACITY
Number of data structures allocated to the ObsGateway pool
Structs
ObsGateway_STRUCT_SIZE
Number of bytes in each struct of the ObsGateway pool
Bytes
ObsGateway_USED
Number of structs in the ObsGateway pool that are currently being used
Structs
PENDING_FIBERS
Number of fibers pending for external events, such as a network packet or SSD response. Upon such an external event, they change state to scheduled fibers
Fibers
QUEUE_TIME_FLEX_TASK_CYCLES
Queue time of deferred flex tasks
Cycles/Sec
RELENTLESS_CYCLES
Number of cycles spent in relentless fibers
Cycles/Sec
RELENTLESS_FIBERS
Number of relentless fibers that are ready to run and eager to get CPU cycles
Fibers
SCHEDULED_FIBERS
Number of current fibers that are ready to run and eager to get CPU cycles
Fibers
SLEEPY_FIBERS
Number of SLEEPY fibers
Fibers
SLEEPY_RPC_SERVER_FIBERS
Number of SLEEPY RPC server fibers
Sleepy fiber detections
SSD_CAPACITY
Number of data structures allocated to the SSD pool
Structs
SSD_STRUCT_SIZE
Number of bytes in each struct of the SSD pool
Bytes
SSD_USED
Number of structs in the SSD pool that are currently being used
Structs
STEP_CYCLES
Histogram of time spent in a fiber
Fiber steps
THREAD_POOL_DEFERRED_TASK_CYCLES
Number of cycles thread pool spent running deferred tasks
Cycles/Sec
THREAD_POOL_FLEX_TASK_CYCLES
Number of cycles thread pool spent running flex tasks
Cycles/Sec
THREAD_POOL_QUEUED_FLEX_TASKS_AVG_RUNTIME
Average runtime of deferred flex tasks run on thread pool
Cycles
THREAD_POOL_TASK_INVOCATIONS
Number of tasks run by thread pool
Invocations/Sec
TIMER_CALLBACKS
Current number of timer callbacks
Callbacks
TOTAL_FIBERS_COUNT
Number of fibers
Fibers
TimedCallback_CAPACITY
Number of data structures allocated to the TimedCallback pool
Structs
TimedCallback_STRUCT_SIZE
Number of bytes in each struct of the TimedCallback pool
Bytes
TimedCallback_USED
Number of structs in the TimedCallback pool that are currently being used
Structs
UploadFileInfo_CAPACITY
Number of data structures allocated to the UploadFileInfo pool
Structs
UploadFileInfo_STRUCT_SIZE
Number of bytes in each struct of the UploadFileInfo pool
Bytes
UploadFileInfo_USED
Number of structs in the UploadFileInfo pool that are currently being used
Structs
networkBuffers_CAPACITY
Number of data structures allocated to the networkBuffers pool
Structs
networkBuffers_USED
Number of structs in the networkBuffers pool that are currently being used
Structs
rdmaNetworkBuffers_CAPACITY
Number of data structures allocated to the rdmaNetworkBuffers pool
Structs
rdmaNetworkBuffers_USED
Number of structs in the rdmaNetworkBuffers pool that are currently being used
Structs
SSD
Type
Description
Units
CLEAN_CHUNK_SKIPPED
Number of clean chunks skips
Chunks
DRIVE_ACTIVE_IOS
The number of in-flight IO against the SSD during sampling
IOs
DRIVE_AER_RECEIVED
Number of AER reports
reports
DRIVE_CANCELLED_COMPLETED_BLOCKS
Drive cancelled completed blocks
Blocks/Sec
DRIVE_CANCELLED_NOT_SUBMITTED_BLOCKS
Drive cancelled not submitted blocks
Blocks/Sec
DRIVE_COMPLETED_OVER_COUNT
Drive completed count > 1 detected
Occurrences
DRIVE_E2E_CORRECTION_COUNT
Drive E2E correction count
Error Count
DRIVE_ENDURANCE_USED
Drive endurance percentage used
%
DRIVE_FORFEITS
Number of IOs forfeited due to lack of memory buffers
Operations/Sec
DRIVE_IDLE_CYCLES
Number of cycles spent in idle
Cycles/Sec
DRIVE_IDLE_TIME
Percentage of the CPU time not used for handling I/Os
%
DRIVE_IO_OVERLAPPED
Number of overlapping IOs
Operations
DRIVE_IO_TOO_LONG
Number of IOs that took longer than expected
Operations/Sec
DRIVE_LATENCY
Measure the latencies up to 5ms (higher latencies are grouped)
Requests
DRIVE_LOAD
Drive Load at sampling time
Load
DRIVE_MAX_ERASE_COUNT
Drive maximum block erase count
Erase Count
DRIVE_MEDIA_BLOCKS_READ
Blocks read from the SSD media
Blocks/Sec
DRIVE_MEDIA_BLOCKS_WRITE
Blocks are written to the SSD media
Blocks/Sec
DRIVE_MEDIA_ERRORS
SSD Media Errors
IO/Sec
DRIVE_MIN_ERASE_COUNT
Drive minimum block erase count
Erase Count
DRIVE_NON_MEDIA_ERRORS
SSD Non-Media Errors
IO/Sec
DRIVE_PCI_CORRECTABLE_ERROR_COUNT
Drive PCI Correctable error count
Error Count
DRIVE_PCI_INACCESSIBLE
Number of PCI Inaccessible errors detected
Count
DRIVE_PCI_LINK_RETRAIN_COUNT
Drive PCI link retrain count
Error Count
DRIVE_PENDING_IOS
The number of IOs waiting to start executing during sampling
IOs
DRIVE_PUMPED_IOS
Number of requests returned in a pump
Pumps
DRIVE_PUMPS_DELAYED
Number of Drive pumps that got delayed
Operations/Sec
DRIVE_PUMPS_SEVERELY_DELAYED
Number of Drive pumps that got severely delayed
Operations/Sec
DRIVE_PUMP_LATENCY
Latency between SSD pumps
Microseconds
DRIVE_READ_LATENCY
Drive Read Execution Latency
Microseconds
DRIVE_READ_OPS
Drive Read Operations
IO/Sec
DRIVE_READ_RATIO_PER_SSD_READ
Drive Read OPS Per SSD Request
Ratio
DRIVE_REMAINING_IOS
Number of requests still in the drive after a pump
Pumps
DRIVE_REMAINING_SPARES
Drive remaining spares
%
DRIVE_REQUEST_BLOCKS
Measure drive request size distribution
Requests
DRIVE_SOFT_ECC_COUNT
Drive Soft ECC Error Count
Error Count
DRIVE_SSD_PUMPS
Number of drive pumps that resulted in the data flow from/to drive
Pump/Sec
DRIVE_UNALIGNED_IOS
Drive unaligned IOs count
Error Count
DRIVE_UNCORRECTABLE_READ_COUNT
Drive uncorrectable read count
Error Count
DRIVE_UTILIZATION
Percentage of time the drive had an active IO submitted to it
%
DRIVE_WAF_INTERVAL
Drive Interval write amplification
Factor
DRIVE_WAF_LIFETIME
Drive lifetime write amplification
Factor
DRIVE_WRITE_LATENCY
Drive Write Execution Latency
Microseconds
DRIVE_WRITE_OPS
Drive Write Operations
IO/Sec
DRIVE_WRITE_RATIO_PER_SSD_WRITE
Drive Write OPS Per SSD Request
Ratio
DRIVE_XOR_RECOVERY_COUNT
Drive XOR recovery count
Error Count
NVKV_CHUNK_OUT_OF_SPACE
Number of failed attempts to allocate a stripe in an NVKV chunk
Attempts/Sec
NVKV_INVALIDATOR_MATCHED
Number of NVKV invalidators matching the data
Attempts/Sec
NVKV_OUT_OF_CHUNKS
Number of failed attempts to allocate an NVKV chunk
Attempts/Sec
NVKV_OUT_OF_SUPERBLOCK_ENTRIES
Number of failed attempts to allocate a superblock NVKV entry
Attempts/Sec
NVME_NAMESPACE_CAPACITY
NVMe namespace capacity
Blocks
NVME_NAMESPACE_SIZE
The size of the NVMe namespace
Blocks
NVME_NAMESPACE_UTILIZATION
NVMe namespace utilization
Blocks
NVME_SMART_AVAILABLE_SPARE_THRESHOLD
Normalized percentage of the available spare falls below the threshold
%
NVME_SMART_AVAILABLE_SPARE
Normalized percentage when the available spare falls below the threshold
%
NVME_SMART_COMPOSITE_TEMP
Current composite temperature of the container in Kelvins
Kelvin
NVME_SMART_CONTROLLER_BUSY_TIME
The duration the controller is busy with I/O commands
Minutes
NVME_SMART_CRITICAL_COMPOSITE_TEMP_TIME
The time spent in critical composite temperature state
Minutes
NVME_SMART_CRITICAL_WARNING
Critical warnings regarding the drive controller state
BitFields
NVME_SMART_DATA_UNITS_READ
The number of 512-byte data units the server has read from the controller (in millions)
Count
NVME_SMART_DATA_UNITS_WRITTEN
The number of 512-byte data units the host has written to the controller (in millions)
Count
NVME_SMART_ERROR_LOG_ENTRIES
The total number of Error Information log entries over the controller's lifetime
Occurrences
NVME_SMART_HOST_READ_CMDS
The number of read commands completed by the controller
Occurrences
NVME_SMART_HOST_WRITE_CMDS
The number of write commands completed by the controller
Occurrences
NVME_SMART_MEDIA_ERRORS
The number of unrecovered data integrity errors detected by the controller
Occurrences
NVME_SMART_POWER_CYCLES
The number of power cycles
Occurrences
NVME_SMART_TEMP_SENSOR_1
The current temperature reported by the temperature sensor 1
Kelvin
NVME_SMART_TEMP_SENSOR_2
The current temperature reported by the temperature sensor 2
Kelvin
NVME_SMART_TEMP_SENSOR_3
The current temperature reported by the temperature sensor 3
Kelvin
NVME_SMART_TEMP_SENSOR_4
The current temperature reported by the temperature sensor 4
Kelvin
NVME_SMART_TEMP_SENSOR_5
The current temperature reported by the temperature sensor 5
Kelvin
NVME_SMART_TEMP_SENSOR_6
The current temperature reported by the temperature sensor 6
Kelvin
NVME_SMART_TEMP_SENSOR_7
The current temperature reported by the temperature sensor 7
Kelvin
NVME_SMART_TEMP_SENSOR_8
The current temperature reported by the temperature sensor 8
Kelvin
NVME_SMART_THERMAL_MGMT_TEMP1_TRANSITION_CNT
The number of times the controller entered lower active power states due to thermal management
Occurrences
NVME_SMART_THERMAL_MGMT_TEMP2_TRANSITION_CNT
The number of times the controller entered lower active power states due to thermal management
Occurrences
NVME_SMART_TOTAL_THERMAL_MGMT_TEMP1_TIME
The total time the controller spent in lower power states for thermal management temperature 1
Seconds
NVME_SMART_TOTAL_THERMAL_MGMT_TEMP2_TIME
The total time the controller spent in lower power states for thermal management temperature 2
Seconds
NVME_SMART_UNSAFE_SHUTDOWNS
The number of unsafe shutdown events
Occurrences
NVME_SMART_USED_PERCENTAGE
Vendor-specific estimate of the percentage of NVM subsystem life used
%
NVME_SMART_WARNING_COMPOSITE_TEMP_TIME
The time spent in warning composite temperature state
Minutes
SSDS_IOS
IOs performed on the SSD service
IO/Sec
SSDS_IO_ERRORS
IO errors on the SSD service
Blocks/Sec
SSD_BLOCKS_READ
Number of blocks read from the SSD service
Blocks/Sec
SSD_BLOCKS_WRITTEN
Number of blocks written to the SSD service
Blocks/Sec
SSD_CHUNKS_IN_USE
Number of allocated chunks
Chunks
SSD_CHUNK_ALLOCS_TRIMMED
Number of chunk allocations from the trimmed queue
Chunks
SSD_CHUNK_ALLOCS_UNTRIMMED
Number of chunk allocations from the untrimmed queue
Chunks
SSD_CHUNK_ALLOCS
Number of chunk allocations
Chunks
SSD_CHUNK_FREES
Number of chunk frees
Chunks
SSD_CHUNK_FREE_TRIMMED
Number of free trimmed chunks
Chunks
SSD_CHUNK_FREE_UNTRIMMED
Number of free untrimmed chunks
Chunks
SSD_CHUNK_TRIMS
Number of trims performed
Chunks
SSD_E2E_BAD_CSUM
End-to-End checksum failures
IO/Sec
SSD_READ_ERRORS
Errors in reading blocks from the SSD service
Blocks/Sec
SSD_READ_LATENCY
Avg. latency of read requests from the SSD service
Microseconds
SSD_READ_REQS_LARGE_NORMAL
Number of large normal read requests from the SSD service
IO/Sec
SSD_READ_REQS
Number of read requests from the SSD service
IO/Sec
SSD_SCRATCH_BUFFERS_USED
Number of scratch blocks used
Blocks
SSD_TRIM_TIMEOUTS
Number of trim timeouts
Timeouts
SSD_WRITES_REQS_LARGE_NORMAL
Number of large normal priority write requests to the SSD service
IO/Sec
SSD_WRITES
Number of write requests to the SSD service
IO/Sec
SSD_WRITE_ERRORS
Errors in writing blocks to the SSD service
Blocks/Sec
SSD_WRITE_LATENCY
Latency of writes to the SSD service
Microseconds
Scrubber
Type
Description
Units
BLOCK_CONSISTENCY_CHECKS
Number of blocks that were checked for consistency against their block-used-state
Blocks/Sec
BLOCK_CONSISTENCY_CHECK_LATENCY
Average latency of checking block consistency
Micros
CLEANED_CHUNKS
Number of chunks that were cleaned by the scrubber
Chunks/Sec
DEGRADED_READS
Number of degraded reads for scrubbing
Requests/Sec
FALSE_USED_CHECK_LATENCY
Average latency of checking false used per block
Micros
FALSE_USED_EXTRA_NOTIFIED
Number of blocks that were notified as used by the mark-extra-used mechanism
Blocks/Sec
INTERRUPTS
Number of scrubs that were interrupted
Occurences/Sec
NETWORK_BUDGET_WAIT_LATENCY
Average latency of waiting for our network budget
Micros
NOT_REALLY_DIRTY_BLOCKS
Number of marked dirty blocks that ScrubMissingWrites found were clean
Blocks/Sec
NUM_COPY_DISCARDED_BLOCKS
Number of copied blocks that were discarded
Blocks/Sec
NUM_COPY_DISCARDS
Number of times we discarded scrubber copy work
Occurences/Sec
NUM_INVENTED_STRIPES_DISCARDS
Number of times we discarded all scrubber work due to invented stripes
Occurences/Sec
NUM_INVENTED_STRIPES_DISCARD_BLOCKS
Number of blocks that were discarded due to invented stripes
Blocks/Sec
NUM_SCRUBBER_DISCARD_INTERMEDIATES
Number of times we discarded all intermediate scrubber work
Occurences/Sec
NUM_SMW_DISCARDED_BLOCKS
Number of SMW'd blocks that were discarded
Blocks/Sec
NUM_SMW_DISCARDS
Number of times we discarded scrubber SMW work
Occurences/Sec
NUM_STRIPE_SKIPPED_NOT_FULLY_READ
Number of stripes skipped since stripe is not fully read
Occurrences
PLACEMENT_SELECTION_LATENCY
Average latency of scrubbed placement selection
Micros
RAID_PLACEMENT_SCANS_COMPLETED
Number of placement scan completions
Occurences
READS_CALLED
Number of blocks that were read
Blocks/Sec
READ_BATCH_SOURCE_BLOCKS
Number of source blocks read per batch
Batches
READ_BLOCKS_LATENCY
Average latency of read blocks
Micros
RELOCATED_BLOCKS
Number of blocks that were relocated for eviction
Blocks/Sec
RELOCATE_BLOCKS_LATENCY
Average latency of relocating blocks
Micros
RETRUSTED_UNPROTECTED_DIRTY_BLOCKS
Number of dirty blocks that ScrubMissingWrites retrusted because they were unprotected
Blocks/Sec
REWRITTEN_DIRTY_BLOCKS
Number of dirty blocks that ScrubMissingWrites rewrote to clean them
Blocks/Sec
SCAN_LIKELY_LEAKED_BLOCKS
Number of free blocks encountered during a scan that was marked as KnownUsed in the RAID
Occurences
SCRUB_BATCHES_LATENCY
Average latency of scrub batches
Millis
SCRUB_FALSE_USED_FAILED_READS
Number of blocks that we failed to read for scrub-false-used
Blocks/Sec
SCRUB_FALSE_USED_FAILED
Number of placements we failed to fully scrub-false-used
Occurences/Sec
SCRUB_FALSE_USED_PLACEMENTS
Number of placements we finished scrub-false-used
Occurences/Sec
SCRUB_FALSE_USED_WAS_UNPROTECTED
Number of blocks that were falsely marked used and unprotected
Blocks/Sec
SCRUB_IN_FLIGHT_CORRUPTION_DETECTED
Number of in-flight corruptions detected when scrubbing
Occurences
SCRUB_PREPARATION_FAILED
Number of times we failed to prepare() a task and aborted scrub of placement
Occurences/Sec
SFU_CHECKS
Number of blocks that were scrubbed-false-used
Blocks/Sec
SFU_CHECK_FREE
Number of blocks that were detected as false-used and freed
Blocks/Sec
SFU_CHECK_SECONDARY
Number of blocks that were detected as secondary
Blocks/Sec
SFU_CHECK_USED_CKSUM_ERR
Number of blocks that were detected as used with checksum error
Blocks/Sec
SFU_CHECK_USED
Number of blocks that were detected as used
Blocks/Sec
SFU_FREE_STRIPES
Number of free stripes that were scrubbed-false-used
Stripes/Sec
SFU_FREE_STRIPE_LATENCY
Average latency of handling a read of a free stripe
Micros
SFU_USED_STRIPES
Number of used stripes that were scrubbed-false-used
Stripes/Sec
SFU_USED_STRIPE_LATENCY
Average latency of handling a read of a used stripe
Micros
SOURCE_READS
Number of source/committed superset blocks directly read by the scrubber
Blocks/Sec
STRIPE_DATA_IS_BLOCK_USED_LATENCY
Average latency of isBlockUsed during stripe verification
Micros
STRIPE_DATA_IS_BLOCK_USED
Number of isBlockUsed during stripe verification
Blocks/Sec
TARGET_COPIED_CHUNKS
Number of chunks that were copied to target by the scrubber
Chunks/Sec
UPDATE_PLACEMENT_INFO_LATENCY
Average latency of updating the placement info quorum
Micros
UPDATE_PLACEMENT_INFO
Number of times we ran updatePlacementInfo
Occurences/Sec
WONT_CLEAN_COPYING
Number of actually dirty blocks that ScrubMissingWrites refused to clean because they will be moved to target anyway
Blocks/Sec
WRITES_CALLED
Number of blocks that were written
Blocks/Sec
WRITE_BATCH_SOURCE_BLOCKS
Number of source blocks to write in batch
Batches
WRITE_BATCH_TARGET_BLOCKS
Number of target blocks to write in batch
Batches
WRITE_BLOCKS_LATENCY
Average latency of writing blocks
Micros
Squelch
Type
Description
Units
BLOCKS_PER_DESQUELCH
Number of squelch blocks per desquelch
Desquelches
EXTENT_DESQUELCHES_NUM
Number of desquelches
Times
EXTENT_SQUELCH_BLOCKS_READ
Number of squelch blocks desquelched
Blocks
HASH_DESQUELCHES_NUM
Number of desquelches
Times
HASH_SQUELCH_BLOCKS_READ
Number of squelch blocks desquelched
Blocks
INODE_DESQUELCHES_NUM
Number of desquelches
Times
INODE_SQUELCH_BLOCKS_READ
Number of squelch blocks desquelched
Blocks
JOURNAL_DESQUELCHES_NUM
Number of desquelches
Times
JOURNAL_SQUELCH_BLOCKS_READ
Number of squelch blocks desquelched
Blocks
MAX_BLOCKS_WITH_TEMPORAL_SQUELCH_ITEMS_IN_BUCKET
Number of block with temporal squelch items in bucket
Blocks
MAX_TEMPORAL_SQUELCH_ITEMS_IN_BUCKET
Number temporal squelch items in bucket
Squelch items
ODL_DESQUELCHES_NUM
Number of desquelches
Times
ODL_PAYLOAD_DESQUELCHES_NUM
Number of desquelches
Times
ODL_PAYLOAD_SQUELCH_BLOCKS_READ
Number of squelch blocks desquelched
Blocks
ODL_SQUELCH_BLOCKS_READ
Number of squelch blocks desquelched
Blocks
REGISTRY_L1_DESQUELCHES_NUM
Number of desquelches
Times
REGISTRY_L1_SQUELCH_BLOCKS_READ
Number of squelch blocks desquelched
Blocks
REGISTRY_L2_DESQUELCHES_NUM
Number of desquelches
Times
REGISTRY_L2_SQUELCH_BLOCKS_READ
Number of squelch blocks desquelched
Blocks
SPATIAL_SQUELCH_DESQUELCHES_NUM
Number of desquelches
Times
SPATIAL_SQUELCH_SQUELCH_BLOCKS_READ
Number of squelch blocks desquelched
Blocks
SUPERBLOCK_DESQUELCHES_NUM
Number of desquelches
Times
SUPERBLOCK_SQUELCH_BLOCKS_READ
Number of squelch blocks desquelched
Blocks
TEMPORAL_SQUELCH_DESQUELCHES_NUM
Number of desquelches
Times
TEMPORAL_SQUELCH_SQUELCH_BLOCKS_READ
Number of squelch blocks desquelched
Blocks
Statistics
Type
Description
Units
AVAILABLE_HOST_MEMORY_MB
Amount of Free Memory
MB
GATHER_FROM_NODE_LATENCY_NET
Time spent on responding to a stats-gathering request (not including metadata)
Seconds/Sec
GATHER_FROM_NODE_LATENCY
Time spent responding to a stats-gathering request (not including metadata)
Seconds/Sec
GATHER_FROM_NODE_SLEEP
Time spent in-between responding to a stats-gathering request (not including metadata)
Seconds/Sec
TIMES_QUERIED_STATS
Number of times the process queried other processes for stats
Times
TIMES_QUERIED
Number of times the process was queried for stats (not including metadata)
Times
Last updated