Set up a Data Services container for background tasks
Efficiently manage resource-intensive tasks with at least one Data Services container for improved performance and reliability.
The Data Services container runs tasks in the background, particularly those that can be resource-intensive. At present, it runs the task. In upcoming releases, it will handle additional tasks that consume significant resources.
Running these tasks in the background ensures your CLI remains accessible and responsive without consuming compute resources. This strategy enhances performance, efficiency, and scalability when managing quotas. If a task is interrupted, it automatically resumes, providing reliability.
If the Data Services container is not operational, the quota coloring task reverts to the previous implementation and runs in a single process. This could result in the CLI hanging for an extended period. Therefore, ensuring the Data Services container runs is crucial to prevent this situation.
To improve data service performance, you can set up multiple Data Service containers, one per WEKA server.
After setting up the Data Service container, you can manage it like any other container within the cluster. If there’s a need to adjust its resources, use the weka cluster container resources or weka local resources commands. For more details, see Expand specific resources of a container.
Before you begin
Ensure the server where you’re adding this container has sufficient memory available:
3.5 GB if no dedicated core is specified.
5.5 GB if a dedicated core is specified.
The Data Service containers require a persistent 22 GB filesystem for intermediate global configuration data. Do one of the following:
If a configuration filesystem for the protocol containers exists (typically named
.config_fs), use it and expand its size by 22 GB. See Dedicated filesystem requirement for cluster-wide persistent protocol configurationsIf a configuration filesystem does not exist, create a dedicated 22 GB configuration filesystem for the Data Service containers.
Set the Data Service global configuration. Run the following command:
weka dataservice global-config set --config-fs <configuration filesystem name>Example:
weka dataservice global-config set --config-fs .config_fsProcedure
Set up the Data Services container: Run the following command:
weka local setup container --name <container_name> --base-port <base-port> --join-ips <management-ip> --only-dataserv-cores --allow-mix-settingParameters:
name*
The Data Services container name. Setdataserv0 to avoid confusion.
only-dataserv-cores*
Creates a Data Services container. This parameter is mandatory.
base-port
If a base-port is not specified, the Data Services container may still initialize as it attempts to allocate an available port range and could succeed. However, for optimal operation, it is recommended to provide the base port externally.
join-ips*
Specify the management IP of one of the servers in the cluster to join.
management-ips
This is optional. If not provided, it automatically takes the management IP of the server.
memory
Configure the container memory to be allocated for huge pages. It is recommended to set it to 1.5 GB.
allow-mix-setting
This option enables using specified core IDs, even when containers with AUTO core ID allocation run on the same server. It is required if the core allocation is not explicitly specified.
Verify the Data Services container is up: Run
weka local ps.
Verify the Data Services container is visible in the cluster: Run
weka cluster container.
Verify the data services and management processes have joined the cluster: Run
weka cluster process.
Last updated