> For the complete documentation index, see [llms.txt](https://docs.weka.io/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.weka.io/operation-guide/set-up-a-data-services-container-for-background-tasks.md). # Set up a Data Services container for background tasks Configure a Data Services container to run quota coloring and S3 lifecycle background tasks. The Data Services container runs tasks in the background, particularly those that can be resource-intensive. It runs the [Quota Coloring](#user-content-fn-1)[^1] task and the [S3 lifecycle management](#user-content-fn-2)[^2] tasks. Running these tasks in the background ensures your CLI remains accessible and responsive without consuming compute resources. This strategy enhances performance, efficiency, and scalability when managing quotas and S3 lifecycle rules. If a task is interrupted, it automatically resumes, providing reliability. {% hint style="warning" %} If the Data Services container is not operational, the quota coloring task reverts to the previous implementation and runs in a single process. This could result in the CLI hanging for an extended period. Therefore, ensuring the Data Services container runs is crucial to prevent this situation. {% endhint %} To improve data service performance, you can set up multiple Data Service containers, one per WEKA server. After setting up the Data Service container, you can manage it like any other container within the cluster. If there’s a need to adjust its resources, use the `weka cluster container resources` or `weka local resources` commands. For more details, see [Expand specific resources of a container](/operation-guide/expanding-and-shrinking-cluster-resources/expansion-of-specific-resources.md). ## **Set up Data Services container** **Before you begin** 1. Ensure the server where you’re adding this container has sufficient memory available: * 3.5 GB if no dedicated core is specified. * 5.5 GB if a dedicated core is specified. 2. The Data Service containers require a persistent filesystem of at least 50 GiB for intermediate global configuration data. Do one of the following: * If a configuration filesystem for the protocol containers exists (typically named `.config_fs`), use it and expand its capacity by 50 GiB. See [Additional protocol containers](/additional-protocols/additional-protocols-overview.md#dedicated-filesystem-requirement-for-cluster-wide-persistent-protocol-configurations) * If a configuration filesystem does not exist, create a dedicated configuration filesystem of at least 50 GiB for the Data Service containers. On a cluster upgraded from an earlier version, expand an existing Data Services configuration filesystem to at least 50 GiB. 3. Set the Data Service global configuration. Run the following command: ``` weka dataservice global-config set --config-fs ``` Example: ``` weka dataservice global-config set --config-fs .config_fs ``` {% hint style="info" %} By default, the Data Service containers share the core of the Management process. However, if you have enough resources, you can assign a separate core to it. {% endhint %} **Procedure** 1. **Set up the Data Services container:** Run the following command: {% code overflow="wrap" %} ```bash weka local setup container --name --base-port --join-ips --only-dataserv-cores --allow-mix-setting ``` {% endcode %} **Parameters:**
ParameterDescription
name*The Data Services container name. Set dataserv0 to avoid confusion.
only-dataserv-cores*Creates a Data Services container. This parameter is mandatory.
base-portIf a base-port is not specified, the Data Services container may still initialize as it attempts to allocate an available port range and could succeed. However, for optimal operation, it is recommended to provide the base port externally.
join-ips*Specify the management IP of one of the servers in the cluster to join.
management-ipsThis is optional. If not provided, it automatically takes the management IP of the server.
memoryConfigure the container memory to be allocated for huge pages. It is recommended to set it to 1.5 GB.
allow-mix-settingThis option enables using specified core IDs, even when containers with AUTO core ID allocation run on the same server. It is required if the core allocation is not explicitly specified.
Example {% code overflow="wrap" %} ```bash $ weka local setup container --name dataserv0 --base-port 14400 --join-ips 10.108.233.213 --only-dataserv-cores --allow-mix-setting Version 6.0.0 is already downloaded. Created Weka container named dataserv0 Preparing version 6.0.0 of container dataserv0 No net parameter specified, configuring in UDP mode Successfully set up container dataserv0 Starting container Waiting for container to start up Container "dataserv0" is ready (pid = 84273) ``` {% endcode %}
2. **Verify the Data Services container is up**: Run `weka local ps`.
Example ```bash $ weka local ps Container Container ID State Disabled Uptime Persistent Port PID Status Version Recent Failure (5 min) Container Type compute0 9 Running False 0:33:17h True 14300 16944 Ready 6.0.0 weka dataserv0 19 Running False 0:06:35h True 14400 84273 Ready 6.0.0 weka drives0 0 Running False 0:34:50h True 14000 15943 Ready 6.0.0 weka frontend0 17 Running False 0:33:07h True 14200 17453 Ready 6.0.0 weka smbw N/A Running False 0:30:42h True 14000 18612 6.0.0 smbw ```
3. **Verify the Data Services container is visible in the cluster:** Run `weka cluster container`.
Example See `dataserv0` in the last row (ID 19). ```bash $ weka cluster container ID Hostname Name IPs Status Requested Action Release Failure Domain Cores Memory Uptime Recent Failure (5 min) Requested Action Failure 0 DataSphere-0 drives0 10.108.233.213 UP NONE 6.0.0 DOM-000 1 1.46 GiB 0:35:19h 1 DataSphere-1 drives0 10.108.111.114 UP NONE 6.0.0 DOM-001 1 1.46 GiB 0:35:21h 2 DataSphere-2 drives0 10.108.190.195 UP NONE 6.0.0 DOM-002 1 1.46 GiB 0:35:20h 3 DataSphere-3 drives0 10.108.220.165 UP NONE 6.0.0 DOM-003 1 1.46 GiB 0:35:20h 4 DataSphere-4 drives0 10.108.107.173 UP NONE 6.0.0 DOM-004 1 1.46 GiB 0:35:18h 5 DataSphere-5 drives0 10.108.176.219 UP NONE 6.0.0 DOM-005 1 1.46 GiB 0:35:17h 6 DataSphere-3 compute0 10.108.220.165 UP NONE 6.0.0 DOM-003 1 1.41 GiB 0:33:47h 7 DataSphere-5 compute0 10.108.176.219 UP NONE 6.0.0 DOM-005 1 1.41 GiB 0:33:46h 8 DataSphere-4 compute0 10.108.107.173 UP NONE 6.0.0 DOM-004 1 1.41 GiB 0:33:46h 9 DataSphere-0 compute0 10.108.233.213 UP NONE 6.0.0 DOM-000 1 1.41 GiB 0:33:45h 10 DataSphere-1 compute0 10.108.111.114 UP NONE 6.0.0 DOM-001 1 1.41 GiB 0:33:46h 11 DataSphere-2 compute0 10.108.190.195 UP NONE 6.0.0 DOM-002 1 1.41 GiB 0:33:45h 12 DataSphere-1 frontend0 10.108.111.114 UP NONE 6.0.0 DOM-001 1 1.38 GiB 0:33:38h 13 DataSphere-2 frontend0 10.108.190.195 UP NONE 6.0.0 DOM-002 1 1.38 GiB 0:33:37h 14 DataSphere-3 frontend0 10.108.220.165 UP NONE 6.0.0 DOM-003 1 1.38 GiB 0:33:37h 15 DataSphere-4 frontend0 10.108.107.173 UP NONE 6.0.0 DOM-004 1 1.38 GiB 0:33:37h 16 DataSphere-5 frontend0 10.108.176.219 UP NONE 6.0.0 DOM-005 1 1.38 GiB 0:33:37h 17 DataSphere-0 frontend0 10.108.233.213 UP NONE 6.0.0 DOM-000 1 1.38 GiB 0:33:35h 18 DataSphere-LinuxClients-6 default 10.108.169.39 UP NONE 6.0.0 1 1.38 GiB 0:31:32h 19 DataSphere-0 dataserv0 10.108.233.213 UP NONE 6.0.0 0 N/A 0:07:03h ```
4. **Verify the data services and management processes have joined the cluster:** Run `weka cluster process`.
Example See PROCESS IDs 380 and 381. ```bash $ weka cluster process Process Container ID Slot Hostname Container IPs Status Release Roles Network CPU Memory Uptime Recent Failure (5 min) 0 0 0 DataSphere-0 drives0 10.108.233.213 UP 6.0.0 MANAGEMENT UDP 0 N/A 0:35:23h 1 0 1 DataSphere-0 drives0 10.108.62.198 UP 6.0.0 DRIVES DPDK 4 1.46 GiB 0:35:19h 20 1 0 DataSphere-1 drives0 10.108.111.114 UP 6.0.0 MANAGEMENT UDP 0 N/A 0:35:15h 21 1 1 DataSphere-1 drives0 10.108.93.124 UP 6.0.0 DRIVES DPDK 4 1.46 GiB 0:35:10h 40 2 0 DataSphere-2 drives0 10.108.190.195 UP 6.0.0 MANAGEMENT UDP 0 N/A 0:35:15h 41 2 1 DataSphere-2 drives0 10.108.217.168 UP 6.0.0 DRIVES DPDK 4 1.46 GiB 0:35:10h 60 3 0 DataSphere-3 drives0 10.108.220.165 UP 6.0.0 MANAGEMENT UDP 0 N/A 0:35:15h 61 3 1 DataSphere-3 drives0 10.108.199.216 UP 6.0.0 DRIVES DPDK 4 1.46 GiB 0:35:10h 80 4 0 DataSphere-4 drives0 10.108.107.173 UP 6.0.0 MANAGEMENT UDP 0 N/A 0:35:15h 81 4 1 DataSphere-4 drives0 10.108.230.210 UP 6.0.0 DRIVES DPDK 4 1.46 GiB 0:35:10h 100 5 0 DataSphere-5 drives0 10.108.176.219 UP 6.0.0 MANAGEMENT UDP 0 N/A 0:35:14h 101 5 1 DataSphere-5 drives0 10.108.147.66 UP 6.0.0 DRIVES DPDK 4 1.46 GiB 0:35:10h 120 6 0 DataSphere-3 compute0 10.108.220.165 UP 6.0.0 MANAGEMENT UDP 0 N/A 0:33:54h 121 6 1 DataSphere-3 compute0 10.108.94.64 UP 6.0.0 COMPUTE DPDK 6 1.41 GiB 0:33:49h 140 7 0 DataSphere-5 compute0 10.108.176.219 UP 6.0.0 MANAGEMENT UDP 0 N/A 0:33:54h 141 7 1 DataSphere-5 compute0 10.108.213.132 UP 6.0.0 COMPUTE DPDK 6 1.41 GiB 0:33:49h 160 8 0 DataSphere-4 compute0 10.108.107.173 UP 6.0.0 MANAGEMENT UDP 0 N/A 0:33:54h 161 8 1 DataSphere-4 compute0 10.108.143.165 UP 6.0.0 COMPUTE DPDK 6 1.41 GiB 0:33:49h 180 9 0 DataSphere-0 compute0 10.108.233.213 UP 6.0.0 MANAGEMENT UDP 0 N/A 0:33:54h 181 9 1 DataSphere-0 compute0 10.108.197.159 UP 6.0.0 COMPUTE DPDK 6 1.41 GiB 0:33:48h 200 10 0 DataSphere-1 compute0 10.108.111.114 UP 6.0.0 MANAGEMENT UDP 0 N/A 0:33:49h 201 10 1 DataSphere-1 compute0 10.108.112.214 UP 6.0.0 COMPUTE DPDK 6 1.41 GiB 0:33:43h 220 11 0 DataSphere-2 compute0 10.108.190.195 UP 6.0.0 MANAGEMENT UDP 0 N/A 0:33:49h 221 11 1 DataSphere-2 compute0 10.108.35.79 UP 6.0.0 COMPUTE DPDK 6 1.41 GiB 0:33:43h 240 12 0 DataSphere-1 frontend0 10.108.111.114 UP 6.0.0 MANAGEMENT UDP 0 N/A 0:33:44h 241 12 1 DataSphere-1 frontend0 10.108.7.198 UP 6.0.0 FRONTEND DPDK 2 1.38 GiB 0:33:39h 260 13 0 DataSphere-2 frontend0 10.108.190.195 UP 6.0.0 MANAGEMENT UDP 0 N/A 0:33:44h 261 13 1 DataSphere-2 frontend0 10.108.99.123 UP 6.0.0 FRONTEND DPDK 2 1.38 GiB 0:33:39h 280 14 0 DataSphere-3 frontend0 10.108.220.165 UP 6.0.0 MANAGEMENT UDP 0 N/A 0:33:44h 281 14 1 DataSphere-3 frontend0 10.108.118.79 UP 6.0.0 FRONTEND DPDK 2 1.38 GiB 0:33:39h 300 15 0 DataSphere-4 frontend0 10.108.107.173 UP 6.0.0 MANAGEMENT UDP 0 N/A 0:33:44h 301 15 1 DataSphere-4 frontend0 10.108.148.247 UP 6.0.0 FRONTEND DPDK 2 1.38 GiB 0:33:39h 320 16 0 DataSphere-5 frontend0 10.108.176.219 UP 6.0.0 MANAGEMENT UDP 0 N/A 0:33:44h 321 16 1 DataSphere-5 frontend0 10.108.159.245 UP 6.0.0 FRONTEND DPDK 2 1.38 GiB 0:33:39h 340 17 0 DataSphere-0 frontend0 10.108.233.213 UP 6.0.0 MANAGEMENT UDP 0 N/A 0:33:44h 341 17 1 DataSphere-0 frontend0 10.108.200.201 UP 6.0.0 FRONTEND DPDK 2 1.38 GiB 0:33:39h 360 18 0 DataSphere-LinuxClients-6 default 10.108.169.39 UP 6.0.0 MANAGEMENT UDP 0 N/A 0:31:38h 361 18 1 DataSphere-LinuxClients-6 default 10.108.18.155 UP 6.0.0 FRONTEND DPDK 2 1.38 GiB 0:31:34h 380 19 0 DataSphere-0 dataserv0 10.108.233.213 UP 6.0.0 MANAGEMENT UDP 0 N/A 0:07:09h 381 19 1 DataSphere-0 dataserv0 10.108.233.213 UP 6.0.0 DATASERV UDP 1 N/A 0:07:03h ```
## Set up S3 lifecycle task management After setting up the Data Services container, you can enable and configure S3 lifecycle task management to automate object expiration in S3 buckets. #### Enable S3 lifecycle task management Run the following command to enable the S3 lifecycle task manager: ```bash weka dataservice s3-lifecycle-task enable ``` #### Configure S3 lifecycle task settings (optional) You can customize the S3 lifecycle task manager behavior using the following command: ```bash weka dataservice s3-lifecycle-task set [--max-tasks ] [--interval ] ``` **Parameters:**
ParameterDescription
--max-tasks

Maximum number of concurrent S3 lifecycle tasks that can run simultaneously.

Default: 4

--interval

Interval between lifecycle task manager runs.

Accepts time format: 3s, 2h, 4m, 1d, 1d5h, 1w, infinite, or unlimited. Default: 60 seconds

**Example:** Set maximum concurrent tasks to 6 and interval to 5 minutes ```bash weka dataservice s3-lifecycle-task set --max-tasks 6 --interval 5m ``` #### View S3 lifecycle task configuration To view the current S3 lifecycle task manager configuration, run: ```bash weka dataservice s3-lifecycle-task show ``` Example output: ``` S3 Lifecycle Task Manager Status Status: Enabled Max Concurrent Tasks: 4 Interval (seconds): 60 ``` #### Disable S3 lifecycle task management To disable the S3 lifecycle task manager, run: 
weka dataservice s3-lifecycle-task disable
{% hint style="info" %} Disabling the task manager prevents new lifecycle tasks from being scheduled. Any currently running tasks will complete, but no new tasks will start until theS3 lifecycle task manager is re-enabled. {% endhint %} [^1]: **What is quota coloring?** During the procedure of setting or unsetting a directory quota, the Data Services container creates a background task referred to as `QUOTA_COLORING`. This task scans the entire directory tree and assigns the quota ID to each file and directory within the tree. [^2]: S3 lifecycle task management to automate object expiration in S3 buckets. --- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation by asking a question. Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters: ``` GET https://docs.weka.io/operation-guide/set-up-a-data-services-container-for-background-tasks.md?ask=&goal= ``` `ask` is the immediate question: it should be specific, self-contained, and written in natural language. `goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.