Set up a Data Services container for background tasks
Efficiently manage resource-intensive tasks with at least one Data Services container for improved performance and reliability.
The Data Services container runs tasks in the background, particularly those that can be resource-intensive. At present, it runs the task. In upcoming releases, it will handle additional tasks that consume significant resources.
Running these tasks in the background ensures your CLI remains accessible and responsive without consuming compute resources. This strategy enhances performance, efficiency, and scalability when managing quotas. If a task is interrupted, it automatically resumes, providing reliability.
If the Data Services container is not operational, the quota coloring task reverts to the previous implementation and runs in a single process. This could result in the CLI hanging for an extended period. Therefore, ensuring the Data Services container runs is crucial to prevent this situation.
To improve data service performance, you can set up multiple Data Service containers, one per WEKA server.
After setting up the Data Service container, you can manage it like any other container within the cluster. If there’s a need to adjust its resources, use the weka cluster container resources or weka local resources commands. For more details, see Expand specific resources of a container.
Before you begin
Ensure the server where you’re adding this container has a minimum of 5.5 GB of memory available for the container’s use.
The Data Service containers require a persistent 22 GB filesystem for intermediate global configuration data. Do one of the following:
If a configuration filesystem does not exist, create a dedicated 22 GB configuration filesystem for the Data Service containers.
Set the Data Service global configuration. Run the following command:
weka dataservice global-config set --config-fs <configuration filesystem name>
Example:
weka dataservice global-config set --config-fs .config_fs
By default, the Data Service containers share the core of the Management process. However, if you have enough resources, you can assign a separate core to it.
Procedure
Identify the leader IP address: Run the following command:
weka cluster process --leader'. This IP is used in the join-ips parameter
The Data Services container name. Setdataserv0 to avoid confusion.
only-dataserv-cores*
Creates a Data Services container. This parameter is mandatory.
base-port
If a base-port is not specified, the Data Services container may still initialize as it attempts to allocate an available port range and could succeed. However, for optimal operation, it is recommended to provide the base port externally.
join-ips*
The cluster's leader IP address. This parameter is mandatory to join the Data Services container to the cluster.
management-ips
This is optional. If not provided, it automatically takes the management IP of the server.
memory
Configure the container memory to be allocated for huge pages. It is recommended to set it to 1.5 GB.
