Skip to content
For the complete documentation index, see llms.txt. This page is also available as Markdown.

Background tasks

Learn how the WEKA system runs asynchronous and maintenance operations in the background, what limits apply to them, and how to monitor and control them.

Background tasks handle work that must not interfere with serving IO. Examples include checking metadata integrity, uploading and downloading snapshots, detaching an object store, reducing data, and applying a directory quota to existing data.

Resource consumption

The WEKA system limits background tasks to 5% of the overall CPU. When the CPU is idle, background tasks can use more than the configured resources, and release them immediately when needed to serve IO.

Concurrency limits

Two separate limits apply to background tasks:

Queued tasks: the number of tasks that can exist at the same time. A paused or aborted task still occupies a slot. When the queue is full, a new operation fails rather than waits.

Running tasks: the number of queued tasks that perform work at the same time. Additional tasks remain queued until a slot frees.

Both limits apply cluster-wide.

Task group
Maximum queued
Maximum running

Filesystem tasks

16

Set by the task-specific rules below

Directory quota tasks

32

4

If an operation cannot start because the queue is full, it fails with the following message:

Operation cannot start because there are already 32 tasks running

Wait for a running task to finish, then retry.

Task-specific rules

Snapshot upload and download:

  • Only a single local upload can exist concurrently inside a filesystem.

  • Only a single remote upload inside a filesystem can be done concurrently. Local and remote uploads can co-exist.

  • Only a single upload from any filesystem can exist in the same object store bucket, to prevent uploads from slowing each other down.

  • An object store snapshot download operation cannot run at the same time as another snapshot download or upload operation.

Data reduction:

  • Only one data reduction or data defragmentation task runs in the cluster at a time. If either task exists, no new task of either type starts.

  • The cluster starts these tasks automatically. You do not create them.

Snapshot metadata prefetch:

  • When a snapshot is downloaded from the object store, the system automatically prefetches its metadata as the initial step.

Further restrictions exist between different tasks and between multiple tasks of the same type. When a background task does not run because of a restriction, the system provides a relevant message.

Background tasks list

Task name
Description
Possible actions

OBS_DETACH2

Detaches an object store from a filesystem.

Pause, Resume, Abort until the task reaches the CLEAR_MANIFEST phase

STOW_UPLOAD

Uploads a snapshot from a filesystem to an object store bucket.

Pause, Resume, Abort

STOW_DOWNLOAD_FILESYSTEM

Downloads a filesystem from a locator in an object store.

Pause, Resume

STOW_DOWNLOAD_SNAPSHOT

Downloads a snapshot to a filesystem from a locator in an object store. Includes fetching the snapshot metadata and squashing the filesystem.

Pause, Resume

SNAPSHOT_DELETE

Deletes a snapshot of a filesystem.

Pause, Resume

FSCK

Checks metadata integrity.

Pause, Resume, Abort

DATA_REDUCTION

Reduces data.

Pause, Resume, Abort

DATA_DEFRAG

Defragments data.

Pause, Resume, Abort

FILESYSTEM_RECOMPRESS

Rewrites the data of a filesystem to compress or decompress it.

Pause, Resume, Abort

QUOTA_COLORING

Applies or clears a directory quota on a directory that already contains data. Requires an active Data Services container.

Pause, Resume, Abort

Monitor background tasks

List the running background tasks and their status:

Column
Description

Task ID

Identifier of the task. Use it with the pause, resume, and abort commands.

Type

Task type, as listed in the table above.

State

Current state of the task, such as RUNNING, PAUSED, PAUSING, ABORTING, or WAITING.

Phase

Current phase and the total number of phases. A task in the first phase has not started its main work.

Progress

Completion percentage of the current phase.

User Paused

Whether a user paused the task, as opposed to the system.

Description

The operation the task performs, including the filesystem and path where relevant.

Time

How long ago the task started.

Add -v to include the task UID and the throttle percentage.

A task that stays in its first phase with no progress is queued and waiting for a slot. It is not stuck.

Control background tasks

Pause, resume, or abort a task by its ID:

You can pause and resume any background task. Abort is not available for every task type, and for some types it is available only during specific phases. See the Possible actions column in the table above.

Directory quota tasks

Setting or clearing a directory quota on a directory that already contains data starts a QUOTA_COLORING task. The task applies the quota to the existing contents of the directory.

A QUOTA_COLORING task reports two phases:

Phase
Description

PREPARE 0/2

The task is queued and waiting for a slot.

STAMPING 1/2

The task is applying the quota to the existing contents of the directory. Progress advances during this phase.

While the task runs, weka fs quota list reports the quota status as ADDING. The status changes to ACTIVE when the task completes.

The directory remains readable and writable throughout.

Setting a quota on an empty directory does not start a task. The quota applies immediately, because the system accounts for new data as it is written.

To increase the number of concurrent directory quota tasks, contact the Customer Success Team. The setting requires WEKA supervision.

Manage background tasks using the GUI

The GUI includes a Background Tasks page that displays both active and pending tasks, along with details such as their duration since initiation, state, phase, and progress percentage.

If other tasks have higher priority, you can pause and resume tasks as needed.

The Abort action is available for specific tasks, such as checking metadata integrity. However, it is not applicable to tasks like downloading a filesystem or snapshot, squashing a filesystem, or detaching object storage. To terminate these tasks, delete the associated entity.

Procedure:

  1. From the Monitor tab, select Background Tasks.

  2. To pause a task, select Pause (the button will toggle to Resume).

  3. To resume a paused task, select Resume.

  4. To abort a task, select Abort.

  5. To view waiting tasks (pending), toggle the Show Waiting Tasks switch.

Last updated