Background tasks
Learn how the WEKA system runs asynchronous and maintenance operations in the background, what limits apply to them, and how to monitor and control them.
Background tasks handle work that must not interfere with serving IO. Examples include checking metadata integrity, uploading and downloading snapshots, detaching an object store, reducing data, and applying a directory quota to existing data.
Resource consumption
The WEKA system limits background tasks to 5% of the overall CPU. When the CPU is idle, background tasks can use more than the configured resources, and release them immediately when needed to serve IO.
Concurrency limits
Two separate limits apply to background tasks:
Queued tasks: the number of tasks that can exist at the same time. A paused or aborted task still occupies a slot. When the queue is full, a new operation fails rather than waits.
Running tasks: the number of queued tasks that perform work at the same time. Additional tasks remain queued until a slot frees.
Both limits apply cluster-wide.
Filesystem tasks
16
Set by the task-specific rules below
Directory quota tasks
32
4
If an operation cannot start because the queue is full, it fails with the following message:
Operation cannot start because there are already 32 tasks runningWait for a running task to finish, then retry.
Task-specific rules
Snapshot upload and download:
Only a single local upload can exist concurrently inside a filesystem.
Only a single remote upload inside a filesystem can be done concurrently. Local and remote uploads can co-exist.
Only a single upload from any filesystem can exist in the same object store bucket, to prevent uploads from slowing each other down.
An object store snapshot download operation cannot run at the same time as another snapshot download or upload operation.
Data reduction:
Only one data reduction or data defragmentation task runs in the cluster at a time. If either task exists, no new task of either type starts.
The cluster starts these tasks automatically. You do not create them.
Snapshot metadata prefetch:
When a snapshot is downloaded from the object store, the system automatically prefetches its metadata as the initial step.
Background tasks list
OBS_DETACH2
Detaches an object store from a filesystem.
Pause, Resume, Abort until the task reaches the CLEAR_MANIFEST phase
STOW_UPLOAD
Uploads a snapshot from a filesystem to an object store bucket.
Pause, Resume, Abort
STOW_DOWNLOAD_FILESYSTEM
Downloads a filesystem from a locator in an object store.
Pause, Resume
STOW_DOWNLOAD_SNAPSHOT
Downloads a snapshot to a filesystem from a locator in an object store. Includes fetching the snapshot metadata and squashing the filesystem.
Pause, Resume
SNAPSHOT_DELETE
Deletes a snapshot of a filesystem.
Pause, Resume
FSCK
Checks metadata integrity.
Pause, Resume, Abort
DATA_REDUCTION
Reduces data.
Pause, Resume, Abort
DATA_DEFRAG
Defragments data.
Pause, Resume, Abort
FILESYSTEM_RECOMPRESS
Rewrites the data of a filesystem to compress or decompress it.
Pause, Resume, Abort
QUOTA_COLORING
Applies or clears a directory quota on a directory that already contains data. Requires an active Data Services container.
Pause, Resume, Abort
Monitor background tasks
List the running background tasks and their status:
Task ID
Identifier of the task. Use it with the pause, resume, and abort commands.
Type
Task type, as listed in the table above.
State
Current state of the task, such as RUNNING, PAUSED, PAUSING, ABORTING, or WAITING.
Phase
Current phase and the total number of phases. A task in the first phase has not started its main work.
Progress
Completion percentage of the current phase.
User Paused
Whether a user paused the task, as opposed to the system.
Description
The operation the task performs, including the filesystem and path where relevant.
Time
How long ago the task started.
Add -v to include the task UID and the throttle percentage.
A task that stays in its first phase with no progress is queued and waiting for a slot. It is not stuck.
Control background tasks
Pause, resume, or abort a task by its ID:
You can pause and resume any background task. Abort is not available for every task type, and for some types it is available only during specific phases. See the Possible actions column in the table above.
Directory quota tasks
Setting or clearing a directory quota on a directory that already contains data starts a QUOTA_COLORING task. The task applies the quota to the existing contents of the directory.
A QUOTA_COLORING task reports two phases:
PREPARE 0/2
The task is queued and waiting for a slot.
STAMPING 1/2
The task is applying the quota to the existing contents of the directory. Progress advances during this phase.
While the task runs, weka fs quota list reports the quota status as ADDING. The status changes to ACTIVE when the task completes.
The directory remains readable and writable throughout.
Setting a quota on an empty directory does not start a task. The quota applies immediately, because the system accounts for new data as it is written.
Manage background tasks using the GUI
The GUI includes a Background Tasks page that displays both active and pending tasks, along with details such as their duration since initiation, state, phase, and progress percentage.
If other tasks have higher priority, you can pause and resume tasks as needed.
The Abort action is available for specific tasks, such as checking metadata integrity. However, it is not applicable to tasks like downloading a filesystem or snapshot, squashing a filesystem, or detaching object storage. To terminate these tasks, delete the associated entity.
Procedure:
From the Monitor tab, select Background Tasks.
To pause a task, select Pause (the button will toggle to Resume).
To resume a paused task, select Resume.
To abort a task, select Abort.
To view waiting tasks (pending), toggle the Show Waiting Tasks switch.
Last updated