Background tasks

This page describes the management of background tasks running on the WEKA system.‌

The WEKA system performs internal and external asynchronous operations and maintenance tasks in the background using minimal CPU resources, allowing no interference nor starving the WEKA system from serving high-performing IOs.‌

Background tasks include, for example, checking metadata integrity, downloading and uploading snapshots, and detaching an object store.

Adhere to the following considerations:

  • CPU resource consumption: The WEKA system limits these tasks’ CPU resources to 5% of the overall CPU. When the CPU is idle, background tasks can use more than the configured resources but are immediately freed if needed to serve IOs.

  • Concurrent tasks: The maximum number of concurrent tasks is 16, with restrictions such as:

    • Only a single local upload can exist concurrently inside a filesystem.

    • Only a single remote upload inside a filesystem can be done concurrently (but local and remote uploads can co-exist).

    • Only a single upload from any filesystem can exist in the same object store bucket to prevent slowing down each other uploads.

    • Object store snapshot download operation cannot be run simultaneously with other snapshot download or upload operations.

    • A paused or aborted task is counted as part of the maximum number of concurrent tasks.

  • Snapshot metadata prefetch: When a snapshot is downloaded from the object store, the system automatically prefetches its metadata as the initial step.

  • Multiple background tasks can run in parallel: For all tasks, except the QUOTA_COLORING, up to 32 background tasks can run in parallel. A paused or aborted task is also counted as a running background task. For QUOTA_COLORING, dedicated Data Services containers can run up to 16 tasks in parallel.

More restrictions exist between different tasks and multiple tasks of the same type. If a background task does not run due to a restriction, the system provides a relevant message.

Background tasks list

Taks nameTask descriptionPossible actions

OBS_DETACH

Detaching Object Storage <OBS name> from filesystem <fs name>.

Pause, Resume

STOW_UPLOAD

Uploading snapshot <snapshot name> from filesystem <fs name> to <OBS site> object-store bucket <OBS bucket name>.

Pause, Resume, Abort

STOW_DOWNLOAD_FILESYSTEM

Downloading filesystem <fs name> from locator <snapshot locator> in object-store <OBS bucket name>.

Pause, Resume

STOW_DOWNLOAD_SNAPSHOT

Downloading the snapshot <snapshot name> to <fs name> from locator <snapshot locator> in object-store <OBS bucket name>. This task includes two additional internal phases:

  • Fetching the snapshot metadata.

  • Squashing the filesystem.

Pause, Resume

FSCK

Checking metadata integrity.

Pause, Resume, Abort

DATA_REDUCTION

Compressing data.

Pause, Resume, Abort

DATA_REDUCTION_GC

Garbage collection (GC).

Pause, Resume, Abort

QUOTA_COLORING

Modifies every extent in each existing file included in the quota.

Pause, Resume, Abort