About WEKA performance tests

There are three main performance metrics when measuring a storage system's performance:

1. Latency, which is the time from operation initiation to completion

2. The number of different IO operations (read/write/metadata) that the system can process concurrently

3. The bandwidth of data that the system can process concurrently

Each performance metric applies to read operations, write operations, or a mixture of read and write operations.

‌When measuring the WEKA system performance, different produce different performance characteristics. Additionally, client network configuration (using user-space DPDK networking or kernel UDP) significantly affects performance.

The FIO tool

The is a generic open-source storage performance testing tool that can be defined as described . In this documentation, the usage of FIO version 3.20 is assumed.

All FIO testing is done using the client/server capabilities of FIO. This makes multiple-client testing easier since FIO reports aggregated results for all clients under the test. Single-client tests are run the same way to keep the results consistent.

Start the FIO server on every one of the clients:

Run the test command from one of the clients, note, the clients need to be mounted to a WEKA filesystem.

An example of launching a test (sometest) on all clients in a file (clients.txt) using the server/client model:

An example for the clients' file, when running multiple clients:

An example of aggregated test results:

The single-client or aggregated tests differ in the clients participating in the test, as defined in the clients.txt.

MDTest

MDTest is a generic open-source metadata performance testing tool. In this documentation, the usage of version 1.9.3 is assumed.

WEKA client performance tests

Overall, the tests contained on this page are designed to show off the sustainable peak performance of the filesystem. Care has been taken to make sure they are realistic and reproducible.

Where possible, the benchmarks try to negate the effects of caching. For file testing, o_direct calls are used to bypass the client's cache. In the case of metadata testing, each phase of testing uses different clients. Also, between each test, the Linux caches are flushed to ensure all data being accessed is not present in the cache. While applications will often take advantage of cached data and metadata, this testing focuses on the filesystem's ability to deliver data independent of caching on the client.

While we provide below the output of one iteration, we ran each test several times and provided the average results in the following results summary.

Results summary

Single client results

Aggregated cluster results (with multiple clients)

Test read throughput

This test measures the client throughput for large (1MB) reads. The job below tries to maximize the read throughput from a single client. The test utilizes multiple threads, each one performing 1 MB reads.

Job definition

Test output example

In this test output example, results show a bandwidth of 8.95 GiB/s from a single client.

Test write throughput

This test measures the client throughput for large (1MB) writes. The job below tries to maximize the write throughput from a single client. The test utilizes multiple threads, each one performing 1MB writes.

Job definition

Test output example

In this test output example, results show a bandwidth of 6.87 GiB/s.

Test read IOPS

This test measures the ability of the client to deliver concurrent 4KB reads. The job below tries to maximize the system read IOPS from a single client. The test utilizes multiple threads, each one performing 4KB reads.

Job definition

Test output example

In this test output example, results show 390,494 IOPS from a single client.

Test write IOPS

This test measures the ability of the client to deliver concurrent 4KB writes. The job below tries to maximize the system write IOPS from a single client. The test utilizes multiple threads, each one performing 4KB writes.

Job definition

Test output example

In this test output example, results show 288,215 IOPS from a single client.

Test read latency

This test measures the minimal achievable read latency under a light load. The test measures the latency over a single-threaded sequence of 4KB reads across multiple files. Each read is executed only after the previous read has been served.

Job definition

Example of Test Output

In this test output example, results show an average latency of 229 microseconds, where 99.5% of the writes terminated in 334 microseconds or less.

Test write latency

This test measures the minimal achievable write latency under a light load. The test measures the latency over a single-threaded sequence of 4KB writes across multiple files. Each write is executed only after the previous write has been served.

Job definition

Test output example

In this test output example, results show an average latency of 226 microseconds, where 99.5% of the writes terminated in 293 microseconds or less.

Test metadata performance

The test measures the rate of metadata operations (such as create, stat, delete) across the cluster. The test uses 20 million files: it uses 8 clients, and multiple threads per client are utilized (136), where each thread handles 18382 files. It is invoked 3 times and provides a summary of the iterations.

Job definition

Test output example

Run all benchmark tests

If it is preferred to run all the tests sequentially and review the results afterward, follow the instructions below.

Preparation

From each client, create a mount point in /mnt/weka to a Weka filesystem and create there the following directories:

Copy the FIOmaster.txt file to your server and create the clients.txt file with your clients' hostnames.

Run the benchmark

Run the benchmarks using the following commands: