WEKA performance tests

Measure the key performance metrics of a storage system—latency, IOPS, and bandwidth—using standardized testing procedures.

Overview

When evaluating a storage system’s performance, three primary metrics are considered:

• Latency: The time between the initiation and completion of an I/O operation.

• IOPS: The number of I/O operations (read, write, or metadata) that the system can process per second.

• Bandwidth: The amount of data transferred per second during I/O operations.

Each metric applies to read, write, or mixed workloads. Performance characteristics may vary depending on the mount mode and network configuration (for example, user-space DPDK vs. kernel UDP).

It is important to distinguish between single-client and aggregated performance. A single client may be limited by its local resources, so for accurate cluster-wide measurements, run tests from multiple clients simultaneously.

To ensure that results reflect the filesystem’s true capabilities rather than client-side caching, all benchmarks use direct I/O (O_DIRECT) and clear Linux caches between tests.

Test performance with wekatester

Use the wekatester command-line utility to perform manual performance testing across multiple client hosts. The tool runs FIO (Flexible I/O) workloads on client systems, also referred to as compute nodes, that are connected to a shared network filesystem.

This approach enables consistent, reproducible, and comparable (“apples-to-apples”) performance benchmarking across different storage systems.

Unlike previous versions, automatic cluster and client discovery (wekatester -c) is no longer available.

All servers must now be specified manually when running tests.

Before you begin

Ensure that FIO is installed on all client hosts participating in the tests.

FIO is included in most Linux distributions and can typically be installed using your system’s package manager, for example:

dnf install fio
# or
apt install fio

For more information on installation and usage, see FIO documentation.

Procedure

1. Log in to a client with access to the system under test.

2. Clone the tools repository:

   git clone --depth 1 https://github.com/weka/tools.git

1. Navigate to the wekatester directory:

cd tools/wekatester

1. Run the performance test manually using the following syntax:

./wekatester.py -d <directory> [-w <workload>] [--fio-bin <path>] [server ...]

Command properties

All command properties are optional except the server property.

Option	Description
server*	Required. One or more server hostnames or IPs to use as workers.
-d, --directory	Target directory on the workers where test files will be created. The target filesystem must be mounted at this directory or at a parent directory. Default: /mnt/weka.
-w, --workload	Specifies the workload definition directory from the fio-jobfiles subdirectory structure. Default: default Built-in workload options: default: Four-corners test suite covering read/write bandwidth, latency, and IOPS mixed: 70/30 read/write mixed workload patterns You can create custom workload directories under fio-jobfiles/ and reference them with this option. Example: ./wekatester.py -d /mnt/weka -w mixed server1 server2 server3
--fio-bin	Specifies the path to the fio binary on target servers. Default: /usr/bin/fio Use this option when fio is installed in a non-standard location or when you want to use a specific fio version. Example: ./wekatester.py -d /mnt/weka --fio-bin /opt/fio/bin/fio server1 server2
-v, --verbosity	Increases output verbosity for debugging and detailed monitoring. Verbosity levels: -v: Basic verbose output -vv: Detailed verbose output -vvv: Maximum verbosity with debug information
-V, --version	Displays the wekatester version number.

Example default usage

./wekatester server1 server2 server3... 

During execution, wekatester distributes and runs FIO workloads on the specified servers, collects performance data, and summarizes the results.

Example output

The command displays a summary of the performance results, providing a clear overview of the cluster's capabilities.

starting test run for job 011-bandwidthR.job on <hostname> with <n> workers:
    read bandwidth: 9.37 GiB/s
    total bandwidth: 9.37 GiB/s
    average bandwidth: 2.34 GiB/s per host

starting test run for job 012-bandwidthW.job on <hostname> with <n> workers:
    write bandwidth: 7.72 GiB/s
    total bandwidth: 7.72 GiB/s
    average bandwidth: 1.93 GiB/s per host

starting test run for job 021-latencyR.job on <hostname> with <n> workers:
    read latency: 237 us

starting test run for job 022-latencyW.job on <hostname> with <n> workers:
    write latency: 180 us

Raw FIO results are stored as JSON files, for example:

results_2025-10-07_1112.json

These files contain the full FIO output for detailed analysis.

Wekatester FIO job definitions

The wekatester tool uses a standardized set of Flexible I/O (FIO) tester jobs to ensure consistent and comparable results. These job definitions are provided for users who want to review the testing methodology or run the tests manually.

All jobs use a 2G file size for testing consistency.

Read throughput job definition

This job measures the maximum read bandwidth.

[global]
filesize=2G
time_based=1
startdelay=5
exitall_on_error=1
create_serialize=0
filename_format=$filenum/$jobnum
directory=/mnt/weka
group_reporting=1
clocksource=gettimeofday
runtime=30
ioengine=libaio
disk_util=0
direct=1
numjobs=32

[fio-createfiles-00]
blocksize=1Mi
description='pre-create files'
create_only=1

[fio-bandwidthSR-00]
stonewall
description='Sequential Read bandwidth workload'
blocksize=1Mi
rw=read
iodepth=1

Write throughput job definition

This job measures the maximum write bandwidth.

[global]
filesize=2G
time_based=1
startdelay=5
exitall_on_error=1
create_serialize=0
filename_format=$filenum/$jobnum
directory=/mnt/weka
group_reporting=1
clocksource=gettimeofday
runtime=30
ioengine=libaio
disk_util=0
direct=1
numjobs=32

[fio-createfiles-00]
stonewall
blocksize=1Mi
description='pre-create files'
create_only=1

[fio-bandwidthSW-00]
stonewall
description='Sequential Write bandwidth workload'
blocksize=1Mi
rw=write
iodepth=1

Read IOPS job definition

This job measures the maximum read IOPS using a 4k block size.

[global]
filesize=2G
time_based=1
startdelay=5
exitall_on_error=1
create_serialize=0
filename_format=$filenum/$jobnum
directory=/mnt/weka
group_reporting=1
clocksource=gettimeofday
runtime=30
ioengine=libaio
disk_util=0
direct=1
numjobs=64

[fio-createfiles-00]
blocksize=1Mi
description='pre-create files'
create_only=1

[fio-iopsR-00]
stonewall
description='Read iops workload'
iodepth=8
bs=4k
rw=randread

Write IOPS job definition

This job measures the maximum write IOPS using a 4k block size.

[global]
filesize=2G
time_based=1
startdelay=5
exitall_on_error=1
create_serialize=0
filename_format=$filenum/$jobnum
directory=/mnt/weka
group_reporting=1
clocksource=gettimeofday
runtime=30
ioengine=libaio
disk_util=0
direct=1
numjobs=64

[fio-createfiles-00]
blocksize=1Mi
description='pre-create files'
create_only=1

[fio-iopsW-00]
stonewall
description='Write iops workload'
iodepth=8
bs=4k
rw=randwrite

Read latency job definition

This job measures read latency using a 4k block size.

[global]
filesize=2G
time_based=1
startdelay=5
exitall_on_error=1
create_serialize=0
filename_format=$filenum/$jobnum
directory=/mnt/weka
group_reporting=1
clocksource=gettimeofday
runtime=30
ioengine=libaio
disk_util=0
direct=1
numjobs=1

[fio-createfiles-00]
blocksize=1Mi
description='pre-create files'
create_only=1

[fio-latencyR-00]
stonewall
description='Read latency workload'
bs=4k
rw=randread
iodepth=1

Write latency job definition

This job measures write latency using a 4k block size.

[global]
filesize=2G
time_based=1
startdelay=5
exitall_on_error=1
create_serialize=0
filename_format=$filenum/$jobnum
directory=/mnt/weka
group_reporting=1
clocksource=gettimeofday
runtime=30
ioengine=libaio
disk_util=0
direct=1
numjobs=1

[fio-createfiles-00]
blocksize=1Mi
description='pre-create files'
create_only=1

[fio-latencyW-00]
stonewall
description='Write latency workload'
bs=4k
rw=randwrite
iodepth=1

Testing metadata performance with MDTest

MDTest is an open-source tool designed to test metadata performance, measuring the rate of operations such as file creates, stats, and deletes across the cluster.

MDTest uses an MPI framework to coordinate jobs across multiple nodes. The examples shown here assume the use of MDTest version 1.9.3 with MPICH version 3.3.2 (MPITCH documentation).

Procedure

Run the MDTest benchmark from a client machine with access to the WEKA filesystem. The following command runs the test across multiple clients defined in a hostfile. It uses 8 clients with 136 threads each to test the performance on 20 million files.

Job definition

mpiexec -f <hostfile> -np 1088 mdtest-v-N 136i 3 n 18382 -F -u-d /mnt/weka/mdtest

Result example

The following table shows an example summary from three test iterations.

Operation	Max	Min	Mean	Std Dev
File creation	40784.448	40784.447	40784.448	0.001
File stat	2352915.997	2352902.666	2352911.311	6.121
File read	217236.252	217236.114	217236.162	0.064
File removal	44101.905	44101.896	44101.902	0.004
Tree creation	3.788	3.097	3.342	0.316
Tree removal	1.192	1.142	1.172	0.022

Performance test results summary

The following tables show example results from tests run in specific AWS and SuperMicro environments.

Single client results

Benchmark	AWS	SuperMicro
Read Throughput	8.9 GiB/s	21.4 GiB/s
Write Throughput	9.4 GiB/s	17.2 GiB/s
Read IOPS	393,333 ops/s	563,667 ops/s
Write IOPS	302,333 ops/s	378,667 ops/s
Read Latency	272 µs avg.<br>99.5% completed under 459 µs	144.76 µs avg.<br>99.5% completed under 260 µs
Write Latency	298 µs avg.<br>99.5% completed under 432 µs	107.12 µs avg.<br>99.5% completed under 142 µs

Aggregated cluster results (with multiple clients)

Benchmark	AWS	SuperMicro
Read Throughput	36.2 GiB/s	123 GiB/s
Write Throughput	11.6 GiB/s	37.6 GiB/s
Read IOPS	1,978,330 ops/s	4,346,330 ops/s
Write IOPS	404,670 ops/s	1,317,000 ops/s
Creates	79,599 ops/s	234,472 ops/s
Stats	1,930,721 ops/s	3,257,394 ops/s
Deletes	117,644 ops/s	361,755 ops/s