# Configure data catalog ## Data catalog architecture Gain insights into how data catalog components collaboratively function to index and query filesystem metadata at scale. By deploying dedicated catalog services, the system concurrently scans modified files and stores searchable metadata in a centralized index database. #### Catalog components The catalog feature integrates several components within the WEKA cluster to manage and query metadata: * **Data service containers:** The compute units that power catalog services. The catalog requires a minimum of five data service containers: one serves as the coordinator and the others function as workers. * **Difflist:** A service that runs on the data service containers to detect changes in the filesystems. * **Data manager:** The component that manages rolling snapshots used by the difflist to track data changes. * **Index database:** The central repository that receives indexed fields and stores results for metadata queries. * **Index filesystem (.indexfs):** A dedicated filesystem that stores catalog index data. * **Query API and GUI:** Interfaces used to access and visualize the indexed metadata. **Related topics** [Set up a Data Services container for background tasks](/operation-guide/background-tasks/set-up-a-data-services-container-for-background-tasks.md) [Snapshots](/weka-filesystems-and-object-stores/snapshots.md#track-filesystem-changes-with-the-difflist-rest-api) #### Catalog workflow The data catalog maintains synchronization through a structured three-step process: 1. **Snapshot management:** The Data manager creates rolling snapshots of the filesystems to provide a reference point for the Difflist. 2. **Parallel scanning:** The system identifies changed files and scans them in parallel across the data service containers to maintain high performance. 3. **Indexing and storage:** The service indexes common metadata fields and sends the results to the Index database. This data is stored on the **index filesystem** for long-term retention and querying.

Catalog architecture

## Deploy the catalog services Configure the infrastructure and filesystems required to activate catalog services for your data. #### Before you begin * The catalog services require at least five backend servers. Ensure each server has 32 GB of free memory and connectivity on port 14400. * Each server that runs catalog services must also run a frontend container. The catalog feature requires a frontend container and a data service container on the same server. * For optimal operation, ensure a minimum of 500 GB of available storage for the index filesystem (`.indexfs`). See [#sizing-guidelines](#sizing-guidelines "mention"). #### Procedure 1. **Create the index filesystem:** Add the `.indexfs`. ```bash weka fs add .indexfs default 500GB ``` 2. **Deploy data service containers:** Create one container on each backend server. You can use the same name on all servers or increment the name across servers (for example, `dataserv0`, `dataserv1`, and so on). 
sudo weka local setup container \\
   	--name dataservN \\
   	--base-port 14400 \\
   	--join-ips <CLUSTER_LEADER_IP> \\
   	--only-dataserv-cores \\
   	--allow-mix-setting
3. **Initialize the catalog services:** 1. Add the newly created `dataserv` container IDs to the catalog cluster. ```bash weka catalog cluster add .indexfs --containers ,,,, ``` 2. Check if the catalog services are active: ```bash weka catalog cluster status ``` Output example: ```bash SERVICE NAME CONTAINER ID HOSTNAME CONTAINER IP STATUS ROLE catalog-coordinator 23 sphere-3 dataserv0 10.121.43.123 active COORDINATOR catalog-worker-2 25 sphere-2 dataserv0 10.121.74.183 active WORKER catalog-worker-4 22 sphere-4 dataserv0 10.121.101.84 active WORKER catalog-worker-5 21 sphere-5 dataserv0 10.121.10.21 active WORKER catalog-worker-6 24 sphere-6 dataserv0 10.121.97.143 active WORKER ``` 4. **Enable indexing:** 1. Enable the catalog feature on your specified filesystem. Replace `fs-name` with the name of the filesystem you want to index. ```bash weka fs update --index-enabled true ``` 2. View the filesystem listing as seen from the catalog perspective: ```bash weka catalog fs status ``` Output example: ```bash FILESYSTEM INDEXING HAS METADATA SNAPSHOTS LATEST SNAPSHOT OLDEST SNAPSHOT LAST INGEST LAST ERROR catalogtest Enabled Yes 7 cat-ingest-3.2604061605 cat-ingest-3.2604061553 Never - fs1 Enabled Yes 6 cat-ingest-5.2604061605 cat-ingest-5.2604061555 Never - ``` 3. Verify catalog configuration: ```bash weka catalog config show ``` Output example: ```bash Indexing enabled: true Index filesystem: .indexfs (ID: FSId<4>) Coordinator: gokul-catalog-3 (ID: HostId<23>) IP: 10.121.43.123 Port: 14511 Indexing interval: 1d 0:00:00h Retention period: 30d 0:00:00h Max ingest tasks: 1 ``` 5. **Configure index interval and snapshot retention period:** Adjust these settings to match your workload needs. They dictate how often data is indexed and the duration for which point-in-time snapshots are kept. By default, the `--index-interval` is set to 1 day, and the `--retention-period` is 30 days. Use the following command to update these configurations: ```shell weka catalog config update --index-interval