> For the complete documentation index, see [llms.txt](https://docs.weka.io/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.weka.io/weka-filesystems-and-object-stores/asynchronous-replication/manage-asynchronous-replication.md). # Manage asynchronous replication Configure, operate, and recover asynchronous replication between NeuralMesh clusters. All procedures require ClusterAdmin privileges. Manage asynchronous replication through this workflow: 1. Prepare both clusters. 2. Link the clusters. 3. Create a replication pair. After you create a pair, use the on-demand procedures to monitor, modify, pause, or remove replication, manage file hydration, fail over to the target cluster, and fail back to the original source cluster. The procedures use the `weka replication link` and `weka replication fs-pair` command groups. ## Prepare the clusters for replication Prepare both clusters for replication. Replication traffic passes through the S3 service of each cluster, over a dedicated replica route. Replication creates no bucket, S3 user, or object store filesystem of its own. **Before you begin** * Ensure the clusters can reach each other over the management network (port 14000 by default) and over the S3 port of each cluster. **Procedure** Run steps 1 to 3 on both clusters. 1. Verify the replication license. Run `weka cluster license` and check that **Cross-Cluster Replication** under **Installed License** shows **Licensed**. To add the entitlement, contact your WEKA account team. 2. Set the cluster name. Run `weka cluster update --cluster-name `. The link is named after the cluster at the other end. 3. Create an S3 cluster with at least one server serving S3. Run `weka s3 cluster add`. See [Manage the S3 cluster using the CLI](/additional-protocols/s3/s3-cluster-management/s3-cluster-management-1.md). 4. Optional: For Selective copy - critical paths (`--copy-path` with specific directories), set up a Data Services container on the **target** cluster. Full data copy (`--copy-path full`) and Selective copy - metadata only (`--copy-path none`) do not need it. See [Set up a Data Services container for background tasks](/operation-guide/set-up-a-data-services-container-for-background-tasks.md). ## Link the clusters Link the two clusters so they can authenticate each other and replicate filesystems. One `weka replication link add` command, run on either cluster, creates the link on both clusters. It authenticates to the other cluster with a cluster admin account on that cluster, pins the TLS certificate the other cluster presents, and creates the replication credentials for both clusters. There are no S3 users or keys to manage for replication. To rotate the credentials of a link, contact the Customer Success Team. **Before you begin** * Complete the preparation on both clusters. * Get the management hostname or IP address of the other cluster, and the username and password of a cluster admin on it. * If the clusters can reach each other only through public addresses, contact the Customer Success Team before you link them. The source cluster hosts the filesystem being replicated. The target cluster receives the replicated filesystem. This procedure runs the link command on the source cluster. **Procedure** 1. On the **target cluster**, display the fingerprint of its TLS certificate, and record the colon-separated hexadecimal value: ```bash weka security tls status ``` 2. On the **source cluster**, link to the target cluster: 
weka replication link add \
  --peer-host <target management host> \
  --username <cluster admin on the target> \
  --fingerprint <target fingerprint>
Provide the password with `--password`, with the `WEKA_PEER_PASSWORD` environment variable, or at the prompt. On success, the command reports the ID of the new link: `Cluster link added with ID .` If you run the command interactively without `--fingerprint`, it displays the fingerprint the other cluster presents and asks you to confirm it. Compare it with the `weka security tls status` output from the other cluster before you continue. If a link to the other cluster already exists on either cluster, the command reports which cluster holds it and makes no change. Remove the existing link, then add it again. See [Remove a replication link](#remove-a-replication-link). **Parameters for weka replication link add**
ParameterDescription
--peer-host*Management hostname or IP address of the cluster to link to.
--username*Cluster admin username on the other cluster.
--passwordPassword for --username. Alternatively, use the WEKA_PEER_PASSWORD environment variable or the interactive prompt.
--fingerprintExpected SHA-256 fingerprint of the TLS certificate of the other cluster, as shown by weka security tls status on that cluster. Required in non-interactive mode unless you use --auto-accept-peer-cert.
--auto-accept-peer-certAccepts the TLS certificate the other cluster presents, without confirming its fingerprint. The certificate is still pinned and checked on every later connection.
--peer-portManagement API port of the other cluster. Default: 14000.
3. Verify the link on either cluster: ```bash weka replication link ``` Confirm that **Connection** shows `connected` and **Pairing** shows `mutual`. Record the link **ID**. You use it to create replication pairs. A link is identified by its ID. Its name follows the name of the other cluster, so two links can have the same name. Every `weka replication link` command, and `weka replication fs-pair add --link-id`, takes the ID. ### Refresh a replication link Refresh a link when the other cluster changes its name, management addresses, or S3 endpoints. Both clusters replace what they hold about each other with what the other cluster now reports. The refresh runs over the existing authenticated link and requires no password. **Procedure** 1. Refresh the link on either cluster: ```bash weka replication link refresh [--peer-host ] [--peer-port ] ``` Use `--peer-host` and `--peer-port` to reach the other cluster for this run when its recorded addresses no longer work. The command does not store these values. If a different cluster answers at the recorded address, the refresh stops. Remove the link and add it again. ## Create a replication pair Create a replication pair between a local filesystem and a filesystem on a linked cluster, and define its replication policy. The replication policy determines the replication interval, which paths are copied proactively, when snapshots become visible on the target, and how many snapshots are retained. **Before you begin** * Choose the copy option and access strategy for the workload. See [Asynchronous replication](/weka-filesystems-and-object-stores/asynchronous-replication.md). For a full data copy, ensure the target capacity is at least the source filesystem size. * Ensure the link to the target cluster shows **Connection** `connected` or `degraded` and **Pairing** `mutual` in `weka replication link`. * Ensure the target cluster has a filesystem group with the same name as the group of the source filesystem. The target filesystem is created in that group. **Procedure** 1. Create the replication pair on the source cluster: ```bash weka replication fs-pair add \ --source-filesystem \ --link-id \ --target-filesystem \ --interval \ [--copy-path ] \ [--access-strategy ] \ [--apply-strategy ] \ [--snapshots-to-keep ] \ [--target-total-capacity ] \ [--now] ``` The target filesystem is created automatically on the target cluster during the first replication cycle. It must not already exist. If it does, the command fails. On success, the command returns the new pair and its ID: ``` ╭────┬──────────────────────────────────────────────────────╮ │ ✅ │ Added replication pair data_fs → data_fs_dest (3). │ ╰────┴──────────────────────────────────────────────────────╯ ``` **Parameters** The following parameters apply to `weka replication fs-pair add`. They do not apply to `weka replication fs-pair`.
ParameterDescription
--source-filesystemName of the local source filesystem.
--link-idID of the replication link to the target cluster, as shown by weka replication link.
--target-filesystemName of the filesystem on the remote cluster. It can be the same as the source filesystem name.
--intervalReplication interval, for example, 5m or 1h. From 5 minutes to 30 days.
--copy-pathSpecifies up to 10 paths to copy proactively. Each path can be up to 1,023 characters long, and all paths together up to 2,048 characters. Separate paths with commas or repeat the option. Use full, all, or / to copy all data. Use none or null to replicate metadata only. Default: metadata-only replication.
--access-strategyControls when a target snapshot becomes accessible. INSTANT_ACCESS (default) exposes the snapshot immediately. Data not copied locally is retrieved on demand. COPY_FIRST exposes the snapshot only after the --copy-path data is local.
--apply-strategyControls when the target applies a replicated snapshot. AUTOMATIC applies the snapshot after the prerequisite phase completes.
--snapshots-to-keepNumber of snapshots to retain, from 2 to 25. Default: 3. Retaining more snapshots requires more storage. Enforced only while the pair is running; see Pause and resume replication.
--target-total-capacityTotal capacity for the target filesystem. Default: same as the source filesystem. A smaller target is allowed for a selective copy. A full copy requires at least the source size.
--nowTriggers the first replication cycle immediately instead of waiting one full interval.
**Examples** Create a full data copy pair for disaster recovery over link 1, replicating every 5 minutes: ```bash weka replication fs-pair add --source-filesystem data_fs --link-id 1 --target-filesystem data_fs_dest --interval 5m --copy-path full --snapshots-to-keep 25 ``` Replicate only selected directories: ```bash weka replication fs-pair add --source-filesystem data_fs3 --link-id 1 --target-filesystem data_fs3_dest --interval 5m --copy-path "/dir1,/dir2" --snapshots-to-keep 10 ``` 2. Verify that the pair is created and running: ```bash weka replication fs-pair ``` ## Monitor replication status Monitor the state, progress, and health of replication pairs and cluster links. ### View replication pairs List the replication pairs and their current status: ```bash weka replication fs-pair ``` The output shows for each pair: * **Role**: `SOURCE` for a pair this cluster drives, or `TARGET` for a pair that another cluster replicates into this one. A `TARGET` row is read-only. To change the pair, run the command on the source cluster. * **State**: The overall state of the pair, such as `RUNNING` or `ERROR`. * **Link**: The name of the linked cluster. * **Interval, Apply, Access, Copy**: The configured replication policy. * **Last Replication**: The timestamp of the last completed replication cycle. Use it to assess the current recovery point. * **Current Status**: The current activity of the pair, such as `COPY`, `IDLE`, or `ERROR`. If a copy task is stuck, the status includes the reason. Each cluster assigns its own pair IDs, so the same pair can have different IDs on the source and the target clusters. For the pair UID, the link ID, and the number of snapshots to keep, run: ```bash weka replication fs-pair -v ``` To customize the output, use the `--output`, `--filter`, and `--sort` options with any of the available columns, including `last-error` and `last-error-time` for troubleshooting. ### Monitor replication interval compliance The replication interval sets the target for each pair. There is no separate setting. The system times each cycle from its start and raises an alert when the cycle overruns the interval by more than one minute. Replication alerts are raised only on the source cluster and are not propagated to the target cluster.
AlertSeverityRaised when the current cycle has been running longer than
Replication behind scheduleMinorThe configured interval, plus one minute
Replication behind scheduleMajorThree times the configured interval, plus one minute
The alert names the filesystem and the linked cluster, and reports the interval, how long the cycle has run, by how much it exceeds the target, and the time of the last completed cycle. Use the alert history as the compliance record. No replication interval alert is raised for a pair that is paused, not running, or still in its first replication cycle. View active alerts: ```bash weka alerts ``` If a replication interval alert fires, inspect the cycle: ```bash weka replication fs-pair --verbose ``` Common causes are a slow or broken connection to the linked cluster, an S3 service problem on the linked cluster, or low free capacity on the target. Repair the connection or the S3 service, or free space. If the source change rate consistently outpaces the connection, lengthen the interval or add bandwidth. ### View replication link health Check the connection and pairing status of each replication link. The status reflects the management endpoints of the other cluster. To check its S3 service, watch the **Last Error** column of the pairs that use the link, and the replication interval alerts. If **Last Error** reports an S3 authentication failure, contact the Customer Success Team. **Procedure** 1. List the replication links: ```bash weka replication link ``` To include the link UID, data IPs, and data port, add `-v`. 2. Check the **Connection** and **Pairing** columns:
ColumnValueMeaning
ConnectionconnectedEvery management endpoint of the other cluster answers.
ConnectiondegradedSome management endpoints of the other cluster answer.
ConnectiondisconnectedNo management endpoint of the other cluster answers.
PairingmutualThe other cluster holds a matching link.
Pairinglocal-onlyThe other cluster holds no matching link. To repair it, remove the link and add it again.
PairingunknownNo management endpoint of the other cluster answers, so the pairing can’t be determined.
## Modify the replication policy Modify the policy of an existing replication pair without recreating it. Run the commands on the source cluster. **Before you begin** Identify the replication pair ID: ```bash weka replication fs-pair ``` **Procedure** 1. Update the pair policy: ```bash weka replication fs-pair update \ [--interval ] \ [--copy-path | --add-copy-path --remove-copy-path ] \ [--access-strategy ] \ [--apply-strategy ] \ [--snapshots-to-keep ] ``` Use the parameter descriptions in the preceding table, with the following additions: * `--copy-path` replaces the entire copy path set. Use `none` or `null` to clear the set. Mutually exclusive with `--add-copy-path` and `--remove-copy-path`. * `--add-copy-path` adds paths to the copy path set. Separate multiple paths with commas or repeat the option. * `--remove-copy-path` removes paths from the copy path set. Separate multiple paths with commas or repeat the option. 2. Verify the updated policy: ```bash weka replication fs-pair ``` **Example** Switch a pair to Selective copy - metadata only and lengthen its interval: ```bash weka replication fs-pair update 3 --copy-path none --interval 6m ``` ## Pause and resume replication Pause a replication pair before maintenance or before removing it, and resume it to continue replication. While a pair is paused, no new snapshot deltas are transferred to the target. On-demand data access on the target continues to work. {% hint style="info" %} Snapshot pruning runs only while a pair is running. A paused pair, or one in an error state, retains all of its snapshots until it resumes, so the `--snapshots-to-keep` limit is not enforced during that time. Expect snapshot count and capacity use to grow while a pair is left paused or unattended in error. {% endhint %} ### Pause a replication pair **Procedure** 1. On the source cluster, pause the pair: ```bash weka replication fs-pair pause ``` ### Resume a replication pair **Procedure** 1. On the source cluster, resume the pair: ```bash weka replication fs-pair resume ``` Replication continues from the last consistent state. ## Manage file hydration on the target Fetch the data of individual files to the target cluster proactively, monitor hydration progress, and release file data back to on-demand mode. With a selective replication policy, file data is retrieved from the source cluster when files are accessed on the target. Use the hydration commands to control this behavior per file: fetch a file before a workload needs it, or release local data to free capacity on the target. ### Fetch a file proactively Fetch a file's data blocks from the source cluster in the background: ```bash weka replication fs-pair fetch [--filesystem ] [--snapshot ] ``` You can run up to five fetch commands concurrently. A sixth concurrent fetch command fails. * `` is a local mount path. Alternatively, specify `--filesystem` and provide a path relative to the filesystem root. * Use `--snapshot` to resolve the path within a snapshot view instead of the live filesystem. ### Monitor hydration progress Fetch and release operations run in the background. Check the hydration status for a file: ```bash weka replication fs-pair hydration status [--filesystem ] [--snapshot ] ``` ### Release file data Return a file's data blocks to on-demand mode to free capacity on the target. The release runs in the background: ```bash weka replication fs-pair release [ ...] ``` By default, background dehydration starts when the target filesystem uses 95% of its SSD capacity. It stops when usage falls to 90%. Each `` is a local mount path. Release several files in one command by listing more than one path. Monitor progress with `weka replication fs-pair hydration status`. This command runs on the container that holds the mount, so it cannot be directed at another cluster with `--HOST`. ## Remove a replication pair Remove a replication pair when you no longer need to synchronize the source and target filesystems, or as part of a failover procedure. {% hint style="warning" %} Removing a pair stops further snapshot replication for the filesystem pair, including the retrieval of data on demand. If the target filesystem was created with a selective replication policy, files that were never hydrated become inaccessible after removal. Verify the hydration state before removal. Pause the pair, then run `weka replication fs-pair` and confirm that **Current Status** shows `IDLE`. {% endhint %} **Before you begin** Identify the replication pair ID on the source cluster: ```bash weka replication fs-pair ``` **Procedure** 1. On the **source** cluster, pause the replication pair. A running pair cannot be removed. A pair in the `ERROR` state can be removed without pausing it: ```bash weka replication fs-pair pause ``` Wait until the pair finishes its current cycle and **Current Status** shows `IDLE`. 2. Remove the pair: ```bash weka replication fs-pair remove [--force] ``` The command removes the pair from both clusters, and reports success only after both are done. It prompts for confirmation. Use `--force` to skip the confirmation prompt, for example, in scripts. If the target cluster is unreachable, the removal fails and the pair stays on both clusters. Retry when the target is reachable again. To remove the pair from the source cluster only, add `--local-only`, and then remove it on the target cluster as well. 3. Verify that the pair is no longer listed: ```bash weka replication fs-pair ``` The target filesystem remains write-protected after the pair is removed. To make it writable, run this command on the target cluster: ```bash weka fs update --access rw ``` Do this only when you intend to promote the target, such as during failover. See [Activate the target cluster during failover](#activate-the-target-cluster-during-failover). ### Remove a pair on the target cluster On the target cluster, `weka replication fs-pair remove` removes an incoming pair (role `TARGET`) only with `--local-only`. Use it when the source cluster is unavailable. The pair does not need to be paused: ```bash weka replication fs-pair remove --local-only ``` If the source cluster is still running, its next replication cycle for the pair moves to the error state. Remove the pair on the source cluster as well. ## Remove a replication link Remove a replication link when replication between the two clusters is no longer required. One command removes the link from both clusters. **Before you begin** Remove every replication pair that uses the link, in both directions. See [Remove a replication pair](#remove-a-replication-pair). Identify the link ID: ```bash weka replication link ``` **Procedure** 1. Remove the link on either cluster: ```bash weka replication link remove [--force] ``` The command prompts for confirmation. Use `--force` to skip the confirmation prompt. 2. Verify on both clusters that the link is gone: ```bash weka replication link ``` If the command reports that the link is in use, a replication pair or a task still uses it, or a filesystem on this cluster still receives a replica over it. Remove the remaining pairs, or wait for the task to finish, and run the command again. To remove the link on this cluster only, add `--local-only`. Use it when the other cluster is unreachable or no longer exists, or when the link exists on only one of the clusters. `--local-only` also removes a link that a target filesystem still receives a replica over. Removing a link does not restore write access to the target filesystems. They stay write-protected until you promote them, as described in [Activate the target cluster during failover](#activate-the-target-cluster-during-failover). ## Activate the target cluster during failover Activate the target filesystem when the source cluster becomes unavailable, and redirect clients to it. {% hint style="warning" %} Failover is a manual procedure. Because replication is asynchronous, the target reflects the last replicated snapshot. Writes made on the source after that snapshot are lost. Except for `COPY_FIRST` pairs with a full data copy, the target can reference data that exists only on the source. Removing the pair makes that data permanently unreadable. {% endhint %} **Before you begin** * Confirm the recovery point. On the source cluster, if it is still reachable, check the **Last Replication** timestamp: ```bash weka replication fs-pair ``` * Confirm that the target contains the data you need: * With `COPY_FIRST` and `--copy-path full`, all data is local. No action is required. * Hydrate data before breaking any other pair type. This includes `INSTANT_ACCESS`, Selective copy - critical paths pairs, and smaller targets. * Run `weka replication fs-pair fetch ` for each path you need on the target. Any file you do not fetch is retrieved on demand: opening or reading it on the target filesystem fetches its data from the source. See [Manage file hydration on the target](#manage-file-hydration-on-the-target). {% hint style="info" %} On the target cluster, `weka replication fs-pair` lists the pair with the role `TARGET`. Its pair ID on the target can differ from its ID on the source. {% endhint %} **Procedure** The steps differ depending on whether the source cluster is still reachable. *Planned failover, source cluster reachable:* 1. On the **source** cluster, pause the pair, wait until **Current Status** shows `IDLE`, and remove it. The removal clears the pair on both clusters: ```bash weka replication fs-pair pause weka replication fs-pair remove ``` *Disaster failover, source cluster unavailable:* 1. On the **target** cluster, remove the incoming pair. The source is unreachable, so the pair is removed on the target only: ```bash weka replication fs-pair remove --local-only ``` *Both cases, to promote the target:* 2. Remove write protection from the target filesystem. On the target cluster: ```bash weka fs update --access rw ``` 3. Mount the filesystem on a client and verify the data before redirecting production traffic to it. When the source cluster returns, its pair moves to the error state on the next cycle. Remove the pair on the source cluster with `weka replication fs-pair remove `. ## Fail back to the original source cluster After a failover, return replication and clients to the original source cluster by creating a replication pair in the reverse direction, from the former target to the original source. Keep the cluster link: one link carries replication in both directions. The new pair starts with a full first replication cycle. **Before you begin** * Confirm that the original source cluster is running. On the former target cluster, run `weka replication link` and check that the link shows **Connection** `connected` or `degraded` and **Pairing** `mutual`. * After a disaster failover, the original source cluster still lists the old pair, in the `ERROR` state. Remove it on that cluster with `weka replication fs-pair remove `. A pair in the `ERROR` state does not need to be paused. After a planned failover, the pair is already removed from both clusters. * Choose the filesystem name on the original source cluster. The replication pair creates this filesystem in its first cycle, so the name must not exist there. Use a new name, or remove the original filesystem first. **Procedure** 1. On the former target cluster, identify the link ID: ```bash weka replication link ``` 2. On the former target cluster, create the pair. Use a full data copy with `COPY_FIRST`, so all data is local on the original source cluster before you move clients back: ```bash weka replication fs-pair add --source-filesystem --link-id --target-filesystem --interval 5m --copy-path full --access-strategy COPY_FIRST ``` 3. Wait until **Last Replication** shows a completed cycle: ```bash weka replication fs-pair ``` 4. Move clients back by following [Activate the target cluster during failover](#activate-the-target-cluster-during-failover), with the original source cluster as the target. --- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation by asking a question. Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters: ``` GET https://docs.weka.io/weka-filesystems-and-object-stores/asynchronous-replication/manage-asynchronous-replication.md?ask=&goal= ``` `ask` is the immediate question: it should be specific, self-contained, and written in natural language. `goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.