Statistics

This page describes the statistics available in the WekaIO system and how to work with them.

Overview

As the WekaIO system runs, it collects hundreds of statistics on system performance. These statistics help in analyzing the WekaIO system performance and determining the source of any problems.

Five different categories of statistics are available for review - Operations (NFS), Operations (Driver), Object Store, SSD and CPU – and when each category is selected, a list of the possible statistics that can be selected is displayed.

By default, the main statistics page displays the last hour of operation, presenting the WekaIO system operation per second on a time axis.

Statistics View Screen

This Statistics view screen offers a number of options to drill-down into the statistics, according to category. Options include:

  • Mousing over the scrollable graph area to view various performance metrics of the WekaIO matrix cluster.

  • Troubleshooting or obtaining a correlation between events and performance (using the top line which provides links to events that occurred).

  • Adding more statistics to the view (using the Statistics menu).

  • Displaying different statistics simultaneously and toggling between them. By default, the graph area shows Ops/sec for the last hour. Using the "Hour, Day, Week" buttons

    at the bottom-right enables changing of the time interval.

  • Displaying, hiding. deleting, and zooming-in on statistics from defined timelines and dates.

  • Bookmarking specific statistics for future reference and sharing with others (using the URL).

Working with Statistics Using the GUI

Viewing Statistics

To view the statistics screen using the GUI, click the statistics button on the left bar:

Statistics View Screen

Adding Statistics

To select the addition of specific statistics, click the + Add Statistics tab on the right-hand side of the Statistics view screen. The statistics menu will be displayed.

Add Statistics Tab
Statistics Menu

Then select the component for which statistics required from the six possible categories. As each component is selected, the list of possible Statistics Names that can be selected changes. It is also possible to searching for a specific statistic by typing the name of the statistic in the Filter field at the top of the menu.

Up to 5 different statistics can be displayed simultaneously. Selecting a metric adds its graph to the Statistics view, together with a selector containing the category and name of the statistics which are displayed according to the appropriate units.

Switching the active unit scale is performed by clicking on one of the inactive units displayed in the left bottom corner of the graph display box.

Hiding/Deleting Statistics

To hide or delete statistics from a graph, mouse-over the selector and click either the Hide or Delete button appearing under the selector.

Example of Hide/Delete Buttons

Specifying a Time Frame

To define a specific period of time (start and end) for the statistics to be displayed, click the From and To selectors appearing in the left corner of the graph display box. Then select the date of the statistics required from the calendar popup and the hours from the right scroller, or by using the up and down arrows that appear when hovering on the time selectors.

From/To Time Selectors
Calendar Popup

Note: It is also possible to change the time period by dragging the graph left or right.

The Auto Refresh setting offers another option for defining a specific period of time. Activation of Auto Refresh is performed by clicking on one of the buttons displayed in the right bottom corner of the graph display box, according to the desired period (hour, day or week). The selected period of time for the statistics will be automatically updated every minute, until Auto Refresh is deactivated by clicking the 'x' button located at the top right of the Auto Refresh tool tip, or by defining a period of time using the time selector.

Auto Refresh Buttons

Obtaining a Summary of Events

To obtain a summary of events that occurred in a specific time period, click the events bubble displayed above the graph. The Events popup will be displayed. Expand the popup in order to obtain a detailed list of events. Click on the icon next to each event to link to the selected event in the Events view screen.

Events Popup

Working with Statistics Using the CLI

Command: weka stats

Use the following command line to obtain statistics definition information: weka stats list-types

Note: This command can be filtered according to names or categories of statistics (when the name or category are defined).

Use the following command line to obtain the current performance status of the hosts: weka stats realtime

Use the following command line to manage filters and read statistics: weka stats --start-time=<start> [--end-time=<end>] [--category=<category>]... [--stat=<stat>]... [--resolution-secs=<secs>] [--accumulated] [--node-ids=<node>...] [--param=<key:val>]... [--no-zeros] [--show-internal] [--per-node] or:

weka stats --interval=<interval> [--category=<category>]... [--stat=<stat>]... [--resolution-secs=<secs>] [--accumulated] [--node-ids=<node>...] [--param=<key:val>]... [--no-zeros] [--show-internal] [--per-node]

Parameters in Command Lines

Name

Type

Value

Limitations

Mandatory

Default

start

String

Start time of the reported period

Valid date and time*

Yes

end

String

End time of the reported period

Valid date and time*

No

Current time

interval**

String

Period of time to be reported

Valid interval in seconds (positive integer number)

Yes

category

String

Specific categories for retrieval of appropirate statistics

Valid existing categories: CPU, Object Store, Operations, Operations (NFS), Operations (Driver), SSD

No

All

stat

String

Statistics names

Valid statistics names

No

All

secs

String

Length of each interval in the reported period

Must be multiples of 60 seconds

No

60 seconds

nodes

String

Node id

Valid node-id

No

All

key:val

String

A pair of key and value, where key is a statistics parameterization type and val is a valid parameterization value for that type

Valid parameterization type and value

No

Notes:

*Refer to Datetime Switches Syntax section in weka --help-syntax for help regarding datetime typed switches.

**Relevant to the second command.

Optional Flags in Command Line

[--accumulated]: Displays accumulated statistics, not rate statistics

[--no-zeros]: Filters results where the value is 0

[--show-internal]: Displays internal statistics

[--per-node]: Does not aggregate statistics across nodes

List of Statistics Collected

This section details the statistics collected to analyze system performance and determine the source of any problems as the WekaIO system runs, according to the following six categories: Operations, Operations (NFS), Operations (Driver), Object Storage, SSD and CPU. When each category is selected, a list of the possible statistics that can be selected is displayed.

Note: All statistics are averaged over 1 second intervals. Consequently, "total" or other aggregates relate to a specific minute.

Operations Statistics

Statistic

Description

ACCESS_LATENCY

Average latency of ACCESS operations, in microseconds

ACCESS_OPS

Number of ACCESS operations per second

COMMIT_LATENCY

Average latency of COMMIT operations, in microseconds

COMMIT_OPS

Number of COMMIT operations per second

CREATE_LATENCY

Average latency of CREATE operations, in microseconds

CREATE_OPS

Number of CREATE operations per second

FILEATOMICOPEN_LATENCY

Average latency of FILEATOMICOPEN operations, in microseconds

FILEATOMICOPEN_OPS

Number of FILEATOMICOPEN operations per second

FILECLOSE_LATENCY

Average latency of FILECLOSE operations, in microseconds

FILECLOSE_OPS

Number of FILECLOSE operations per second

FILEOPEN_LATENCY

Average latency of FILEOPEN operations, in microseconds

FILEOPEN_OPS

Number of FILEOPEN operations per second

FLOCK_LATENCY

Average latency of FLOCK operations, in microseconds

FLOCK_OPS

Number of FLOCK operations per second

FSINFO_LATENCY

Average latency of FSINFO operations, in microseconds

FSINFO_OPS

Number of FSINFO operations per second

GETATTR_LATENCY

Average latency of GETATTR operations, in microseconds

GATATTR_OPS

Number of GETATTR operations per second

LINK_LATENCY

Average latency of LINK operations, in microseconds

LINK_OPS

Number of LINK operations per second

LOOKUP_LATENCY

Average latency of LOOKUP operations, in microseconds

LOOKUP_OPS

Number of LOOKUP operations per second

MKDIR_LATENCY

Average latency of MKDIR operations, in microseconds

MKDIR_OPS

Number of MKDIR operations per second

MKNOD_OPS

Number of MKNOD operations per second

MKNOD_LATENCY

Average latency of MKNOD operations per second

OPS

Total number of operations per second

PATHCONF_LATENCY

Average latency of PATHCONF operations, in microseconds

PATHCONF_OPS

Number of PATHCONF operations per second

READ_BYTES

Number of byte reads per second

READDIR_LATENCY

Average latency of READDIR operations, in microseconds

READDIR_OPS

Number of READDIR operations per second

READ_DURATION

Histogram of read duration, in microseconds

READLINK_LATENCY

Average latency of READLINK operations, in microseconds

READLINK_OPS

Number of READLINK operations per second

READ_LATENCY

Average latency of READ operations, in microseconds

READS

Number of READ operations per second

REMOVE_LATENCY

Average latency of REMOVE operations, in microseconds

REMOVE_OPS

Number of REMOVE operations per second

RENAME_LATENCY

Average latency of RENAME operations, in microseconds

RENAME_OPS

Number of RENAME operations per second

RMDIR_LATENCY

Average latency of RMDIR operations, in microseconds

RMDIR_OPS

Number of RMDIR operations per second

SETATTR_LATENCY

Average latency of SETATTR operations, in microseconds

SETATTR_OPS

Number of SETATTR operations per second

STATFS_LATENCY

Average latency of STATFS operations, in microseconds

STATFS_OP

Number of STATFS operations per second

SYMLINK_LATENCY

Average latency of SYMLINK operations, in microseconds

SYMLINK_OPS

Number of SYMLINK operations per second

THROUGHPUT

Number of byte read/writes per second

UNLINK_LATENCY

Average latency of UNLINK operations, in microseconds

UNLINK_OPS

Number of UNLINK operations per second

WRITES

Number of WRITE operations per second

WRITE_BYTES

Number of byte writes per second

WRITE_DURATION

Histogram of write duration, in microseconds

WRITE_LATENCY

Average latency of WRITE operations, in microseconds

Operations (NFS) Statistics

Statistic

Description

ACCESS_OPS

The number of ACCESS operations per second

ACCESS_LATENCY

The average latency of ACCESS operations, in microseconds

COMMIT_OPS

The number of COMMIT operations per second

COMMIT_LATENCY

The average latency of COMMIT operations, in microseconds

CREATE_OPS

The number of CREATE operations per second

CREATE_LATENCY

The average latency of CREATE operations, in microseconds

FSINFO_OPS

The number of FSINFO operations per second

FSINFO_LATENCY

The average latency of FSINFO operations, in microseconds

GETATTR_OPS

The number of GETATTR operations per second

GETATTR_LATENCY

The average latency of GETATTR operations, in microseconds

LINK_OPS

The number of LINK operations per second

LINK_LATENCY

The average latency of LINK operations, in microseconds

LOOKUP_OPS

The number of LOOKUP operations per second

LOOKUP_LATENCY

The average latency of LOOKUP operations, in microseconds

MKDIR_OPS

The number of MKDIR operations per second

MKDIR_LATENCY

The average latency of MKDIR operations, in microseconds

MKNOD_OPS

The number of MKNOD operations per second

MKNOD_LATENCY

The average latency of MKNOD operations, in microseconds

OPS

The total number of operations per second

PATHCONF_OPS

The number of PATHCONF operations per second

PATHCONF_LATENCY

The average latency of PATHCONF operations, in microseconds

READ_BYTES

The number of bytes read per second

READ_DURATION

Histogram of the time spent performing read operations, in ,microseconds

READ_LATENCY

The average latency of READ operations, in microseconds

READ_SIZES

Histogram of NFS read sizes

READDIR_OPS

The number of READDIR operations per second

READDIR_LATENCY

The average latency of READDIR operations, in microseconds

READLINK_OPS

The number of READLINK operations per second

READLINK_LATENCY

The average latency of READLINK operations, in microseconds

READS

The number of read operations per second

REMOVE_OPS

The number of REMOVE operations per second

REMOVE_LATENCY

The average latency of REMOVE operations, in microseconds

RENAME_OPS

The number of RENAME operations per second

RENAME_LATENCY

The average latency of RENAME operations, in microseconds

SETATTR_OPS

The number of SETATTR operations per second

SETATTR_LATENCY

The average latency of SETATTR operations, in microseconds

STATFS_OPS

The number of STATFS operations per second

STATFS_LATENCY

The average latency of STATFS operations, in microseconds

SYMLINK_OPS

The number of SYMLINK operations per second

SYMLINK_LATENCY

The average latency of SYMLINK operations, in microseconds

THROUGHPUT

The number byte read/write per second

WRITES

The number of WRITE operations per second

WRITE_BYTES

The number of bytes written to a file per second

WRITE_LATENCY

The average latency of WRITE operations, in microseconds

WRITE_DURATION

Histogram of the time spent performing write operations

WRITE_SIZES

Histogram of NFS write sizes

Operations (Driver) Statistics

Statistic

Description

DIRECT_READ_SIZES

Histogram of the sizes of read operations that were not cached, in blocks/second

DIRECT_WRITE_SIZES

Histogram of the sizes of write operations that were not cached, in blocks/second

FILEATOMICOPEN_OPS

The number of FILEATOMICOPEN operations (operations that atomically create and open a file) per second

FILEATOMICOPEN_LATENCY

The average latency of FILEATOMICOPEN operations, in microseconds

FILEOPEN_OPS

The number of FILEOPEN operations per second

FILEOPEN_LATENCY

The average latency of FILEOPEN operations, in microseconds

FILECLOSE_OPS

The number of FILECLOSE operations per second

FILECLOSE_LATENCY

The average latency of FILECLOSE operations, in microseconds

FLOCK_OPS

The number of File LOCK operations per second

FLOCK_LATENCY

The average latency of File LOCK operations, in microseconds

GETATTR_OPS

The number of GETATTR operations per second

GETATTR_LATENCY

The average latency of GETATTR operations, in microseconds

IOCTL_OBS_PREFETCH_OPS

The number of IOCTL OBS PREFETCH operations per second

IOCTL_OBS_PREFETCH_LATENCY

The average latency of IOCTL OBS PREFETCH operations, in microseconds

LINK_OPS

The number of LINK operations per second

LINK_LATENCY

The average latency of LINK operations, in microseconds

LOOKUP_OPS

The number of LOOKUP operations per second

LOOKUP_LATENCY

The average latency of LOOKUP operations, in microseconds

MKNOD_OPS

The number of MKNOD operations per second

MKNOD_LATENCY

The average latency of MKNOD operations, in microseconds

OPS

The total number of operations per second

READS

The number of READ operations per second

READ_BYTES

The number of bytes read per second

READ_DURATION

Histogram of the time spent performing read operations, in microseconds

READ_LATENCY

The average latency of READ operations, in microseconds

READ_SIZES

Histogram of sizes of read operations, in blocks/second

READDIR_OPS

The number of READDIR operations per second

READDIR_LATENCY

The average latency of READDIR operations, in microseconds

RENAME_OPS

The number of file or directory RENAME operations per second

RENAME_LATENCY

The average latency of RENAME operations, in microseconds

RMDIR_OPS

The number of RMDIR operations per second

RMDIR_LATENCY

The average latency of RMDIR operations, in microseconds

READLINK_OPS

The number of READLINK operations per second

READLINK_LATENCY

The average latency of READLINK operations, in microseconds

STATFS_OPS

The number of STATFS operations per second

STATFS_LATENCY

The average latency of STATFS operations, in microseconds

SETATTR_OPS

The number of SETATTR operations per second

SETATTR_LATENCY

The average latency of SETATTR operations, in microseconds

SYMLINK_OPS

The number of SYMLINK operations per second

SYMLINK_LATENCY

The average latency of SYMLINK operations, in microseconds

THROUGHPUT

The total number byte reads/writes per second

UNLINK_OPS

The number of UNLINK operations per second

UNLINK_LATENCY

The average latency of UNLINK operations, in microseconds

WRITES

The number of write operations per second

WRITE_BYTES

The number of bytes written per second

WRITE_DURATION

Histogram of the time spent performing write operations, in microseconds

WRITE_LATENCY

The average latency of WRITE operations, in microseconds

WRITE_SIZES

Histogram of the number of blocks written per second

Object Storage Statistics

Statistic

Description

FAILED_OBJECT_DELETES

The number of FAILED OBJECT DELETE operations (irrespective of the reason for failure) on all object stores per second

FAILED_OBJECT_DOWNLOADS

The number of FAILED OBJECT DOWNLOAD operations (irrespective of the reason for failure) on all object stores per second

FAILED_OBJECT_OPERATIONS

The total number of FAILED OBJECT operations per second

FAILED_OBJECT_UPLOADS

The number of FAILED OBJECT UPLOAD operations (irrespective of the reason for failure) on all object stores per second

OBJECT_DELETE_DURATION

Histogram of the time spent performing OBJECT DELETE operations, in microseconds

OBJECT_DELETE_LATENCY

The latency of OBJECT DELETE operations, in microseconds

OBJECT_DELETES

The number of OBJECT DELETE operations per second

OBJECT_DOWNLOADS

The number of OBJECT DOWNLOAD operations per second

OBJECT_DOWNLOAD_DURATION

Histogram of the time spent performing OBJECT DOWNLOAD operations, in microseconds

OBJECT_DOWNLOAD_LATENCY

The time spent performing OBJECT DOWNLOAD operations, in microseconds

OBJECT_OPERATIONS

The total number of OBJECT operations per second

OBJECT_UPLOADS

The number of OBJECT UPLOAD operations per second

OBJECT_UPLOAD_DURATION

Histogram of the time spent on OBJECT UPLOAD operations, in microseconds

OBJECT_UPLOAD_LATENCY

The latency of OBJECT UPLOAD operations, in microsecondst stores.

OBS_READ_BYTES

The number of READ BYTES read from the object store per second

OBS_WRITE_BYTES

The number of WRITE BYTES sent to the object store per second

READ_BYTES

The number of bytes READ from the object store per second

WRITE_BYTES

The number of bytes sent to the object store per second

SSD Statistics

Statistic

Description

DRIVE_READ_OPS

The rate of DRIVE READ operations, in IO/second

DRIVE_READ_LATENCY

The latency of DRIVE READ operations, in microseconds

DRIVE_WRITE_OPERATIONS

The rate of DRIVE WRITE operations, in IO/second

DRIVE_WRITE_LATENCY

The latency of DRIVE WRITE operations, in microseconds

SSD_BLOCKS_READ

The number of BLOCK READ operations from the SSD service, in blocks/second

SSD_BLOCKS_WRITTEN

The number of BLOCK WRITE operations to the SSD service, in blocks/second

SSD_MEDIA_ERRORS

The number of SSD MEDIA ERRORS, in IO/second

SSD_NON_MEDIA_ERRORS

The number of SSD NON-MEDIA ERRORS, in IO/second

SSD_READ_REQS

The number of read requests from the SSD service, in IO/second

SSD_READ_ERRORS

The number of ERRORS in reading blocks from the SSD service, in blocks/second

SSD_READ_LATENCY

The average latency of READ requests from the SSD, in microseconds

SSD_WRITES

The number of WRITE request operations to the SSD service, in IO/second

SSD_WRITE_LATENCY

The latency of writes to the SSD service, in microseconds

SSD_WRITE_ERRORS

The number of ERRORS in writing blocks to the SSD service, in blocks/second

CPU Statistics

Statistic

Description

CPU_UTILIZATION

Custom statistic on the handling of I/Os, representing the average percentage of CPU time utilized by WekaIO (from the cores used by WekaIO)