Set up a Data Services container for background tasks

Efficiently manage resource-intensive tasks with at least one Data Services container for improved performance and reliability.

The Data Services container runs tasks in the background, particularly those that can be resource-intensive. At present, it runs the Quota Coloring task. In upcoming releases, it will handle additional tasks that consume significant resources.

Running these tasks in the background ensures your CLI remains accessible and responsive without consuming compute resources. This strategy enhances performance, efficiency, and scalability when managing quotas. If a task is interrupted, it automatically resumes, providing reliability.

If the Data Services container is not operational, the quota coloring task reverts to the previous implementation and runs in a single process. This could result in the CLI hanging for an extended period. Therefore, ensuring the Data Services container runs is crucial to prevent this situation.

To improve data service performance, you can set up multiple Data Service containers, one per WEKA server.

After setting up the Data Service container, you can manage it like any other container within the cluster. If there’s a need to adjust its resources, use the weka cluster container resources or weka local resources commands. For more details, see Expand specific resources of a container.

Before you begin

  1. Ensure the server where you’re adding this container has sufficient memory available:

    • 3.5 GB if no dedicated core is specified.

    • 5.5 GB if a dedicated core is specified.

  2. The Data Service containers require a persistent 22 GB filesystem for intermediate global configuration data. Do one of the following:

  3. Set the Data Service global configuration. Run the following command: 

weka dataservice global-config set --config-fs <configuration filesystem name>

Example:

weka dataservice global-config set --config-fs .config_fs

By default, the Data Service containers share the core of the Management process. However, if you have enough resources, you can assign a separate core to it.

Procedure

  1. Set up the Data Services container: Run the following command:

Parameters:

Parameter
Description

name*

The Data Services container name. Setdataserv0 to avoid confusion.

only-dataserv-cores*

Creates a Data Services container. This parameter is mandatory.

base-port

If a base-port is not specified, the Data Services container may still initialize as it attempts to allocate an available port range and could succeed. However, for optimal operation, it is recommended to provide the base port externally.

join-ips*

Specify the management IP of one of the servers in the cluster to join.

management-ips

This is optional. If not provided, it automatically takes the management IP of the server.

memory

Configure the container memory to be allocated for huge pages. It is recommended to set it to 1.5 GB.

Example

  1. Verify the Data Services container is up: Run weka local ps.

Example

  1. Verify the Data Services container is visible in the cluster: Run weka cluster container.

Example

See dataserve0 in the last row (CONTAINER ID 15).

  1. Verify the data services and management processes have joined the cluster: Run weka cluster process.

Example

See PROCESS IDs 300 and 301.

PreviousBackground tasksNextManage background tasks using the GUI

Last updated