The Data Services container runs tasks in the background, particularly those that can be resource-intensive. At present, it runs the task. In upcoming releases, it will handle additional tasks that consume significant resources.
Running these tasks in the background ensures your CLI remains accessible and responsive without consuming compute resources. This strategy enhances performance, efficiency, and scalability when managing quotas. If a task is interrupted, it automatically resumes, providing reliability.
If the Data Services container is not operational, the quota coloring task reverts to the previous implementation and runs in a single process. This could result in the CLI hanging for an extended period. Therefore, ensuring the Data Services container runs is crucial to prevent this situation.
To improve data service performance, you can set up multiple Data Service containers, one per WEKA server.
After setting up the Data Service container, you can manage it like any other container within the cluster. If there’s a need to adjust its resources, use the weka cluster container resources or weka local resources commands. For more details, see Expand specific resources of a container.
Before you begin
Ensure the server where you’re adding this container has a minimum of 5.5 GB of memory available for the container’s use.
The Data Service containers require a persistent 22 GB filesystem for intermediate global configuration data. Do one of the following:
If a configuration filesystem for the protocol containers exists (typically named .config_fs), use it and expand its size by 22 GB. See
If a configuration filesystem does not exist, create a dedicated 22 GB configuration filesystem for the Data Service containers.
Set the Data Service global configuration. Run the following command:
weka dataservice global-config set --config-fs <configuration filesystem name>
Example:
weka dataservice global-config set --config-fs .config_fs
By default, the Data Service containers share the core of the Management process. However, if you have enough resources, you can assign a separate core to it.
Procedure
Identify the leader IP address: Run the following command:
weka cluster process --leader'. This IP is used in the join-ips parameter
Example
$ weka cluster process --leader
PROCESS ID HOSTNAME CONTAINER IPS STATUS RELEASE ROLES NETWORK CPU MEMORY UPTIME LAST FAILURE
60 DatSphere-1 drives0 10.108.234.164 UP 4.3.2 MANAGEMENT UDP N/A 1:21:05h
Set up the Data Services container: Run the following command:
The Data Services container name. Setdataserv0 to avoid confusion.
only-dataserv-cores*
Creates a Data Services container. This parameter is mandatory.
base-port
If a base-port is not specified, the Data Services container may still initialize as it attempts to allocate an available port range and could succeed. However, for optimal operation, it is recommended to provide the base port externally.
join-ips*
The cluster's leader IP address. This parameter is mandatory to join the Data Services container to the cluster.
management-ips
This is optional. If not provided, it automatically takes the management IP of the server.
memory
Configure the container memory to be allocated for huge pages. It is recommended to set it to 1.5 GB.
allow-mix-setting
This option enables using specified core IDs, even when containers with AUTO core ID allocation run on the same server. It is required if the core allocation is not explicitly specified.
Example
$ weka local setup container --name dataserv0 --base-port 14400 --join-ips 10.108.234.164 --only-dataserv-cores --memory 1.5GB --allow-mix-setting
Version 4.3.2 is already downloaded.
Created Weka container named dataserv0
Preparing version 4.3.2 of container dataserv0
No net parameter specified, configuring in UDP mode
Successfully set up container dataserv0
Starting container
Waiting for container to start up
Container "dataserv0" is ready (pid = 66904)
Verify the Data Services container is up: Run weka local ps.
Example
$ weka local ps
CONTAINER STATE DISABLED UPTIME MONITORING PERSISTENT PORT PID STATUS VERSION LAST FAILURE
compute0 Running False 1:21:58h True True 14300 44600 Ready 4.3.2
dataserv0 Running False 44.59s True True 14400 66904 Ready 4.3.2
drives0 Running False 1:22:39h True True 14000 43448 Ready 4.3.2
frontend0 Running False 1:21:15h True True 14200 45680 Ready 4.3.2
Verify the Data Services container is visible in the cluster: Run weka cluster container.
