> For the complete documentation index, see [llms.txt](https://docs.weka.io/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.weka.io/weka-filesystems-and-object-stores/asynchronous-replication.md).

# Asynchronous replication

Learn how asynchronous replication protects your data by synchronizing filesystems directly between two NeuralMesh clusters.

Asynchronous replication is a native, cluster-to-cluster filesystem replication solution that synchronizes data and metadata directly between a source cluster and a target cluster. Replication transfers incremental snapshot deltas of the source filesystem on a configurable interval, without disrupting client access to the source.

The main decision is how much data the target holds: the whole filesystem, only its metadata, or selected directories. Set the replication policy for each pair. For Selective Replication, specify the directories to pre-copy. You can change these settings later without recreating the pair. A Full Data Replication pair needs a target at least the size of the source. A Selective Replication pair can be much smaller, sized for the working set.

## When to use asynchronous replication

### Full data copy

Maintain a complete, incrementally updated filesystem copy on the target cluster. If the source cluster becomes unavailable, manually activate the target filesystem. Recovery time depends on the manual failover procedure.

{% hint style="warning" %}
Because replication is asynchronous, the target lags the source by at least the replication interval. Writes made after the last replicated snapshot are lost on failover.
{% endhint %}

* **Disaster recovery with a replication interval requirement**: The system raises an alert whenever a replication cycle runs past its interval. Use the alert history to verify replication interval compliance. See [Monitor replication interval compliance](https://docs.weka.io/weka-filesystems-and-object-stores/asynchronous-replication/manage-asynchronous-replication.md#monitor-replication-interval-compliance).
* **Ingest site to central compute**: Collection sites replicate continuously to a central cluster that runs the compute, for example, device data feeding a central GPU cluster for training. Data reaches the central cluster without shipped media, and the collection sites need no compute of their own.

### Selective copy - metadata only

Make a large dataset visible on a remote cluster that has far less capacity than the source. The target receives the full filesystem metadata (directories and file hierarchy), and file data is retrieved from the source only when files are accessed (hydration).

* **Remote working set**: Users at a remote site browse the entire namespace but hydrate only the projects in active use. Size the target for the working set, not the full dataset.
* **Bursting to a remote or cloud cluster**: Present the namespace on a cluster that holds none of the data yet. Jobs pull only the files they actually read.

### Selective copy - critical paths

Replicate selected directories proactively while the rest of the namespace remains available on demand. Specify up to 10 directory paths. Each path can be up to 1,023 characters long, and all paths together up to 2,048 characters. Use this when a remote site needs local performance for specific projects and on-demand access to everything else.

* **Pre-staged projects**: Add a project directory to the copy path set before the work starts, so its data is already local when users arrive. See [Modify the replication policy](https://docs.weka.io/weka-filesystems-and-object-stores/asynchronous-replication/manage-asynchronous-replication.md#modify-the-replication-policy).

## Replication architecture

The following diagram shows the components and data flow for an asynchronous replication pair.

<div data-with-frame="true"><figure><img src="https://2945940841-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FU9eAUfjPZz6inqBDXaG4%2Fuploads%2Fgit-blob-2796bc77d3c3bb0103578ed739687c8b48da6352%2FReplication_architecture.png?alt=media" alt=""><figcaption><p>Asynchronous replication architecture</p></figcaption></figure></div>

A replication pair connects a source filesystem with a target filesystem:

* **Cluster link**: Before you can create a replication pair, link the two clusters. One `weka replication link add` command, run on either cluster, creates the link on both. The link authenticates with a cluster admin account on the other cluster, pins its TLS certificate, and carries replication in both directions. Each replication pair replicates in one direction, from its source to its target.
* **Snapshot deltas**: On each replication interval, the system takes a snapshot of the source filesystem and transfers the incremental delta to the target. The minimum interval is 5 minutes.
* **Transport**: Replication uses the S3 infrastructure of the clusters as a transport layer. Data passes through the S3 service of each cluster, over a dedicated replica route.
* **Source and target roles**: The source filesystem remains fully readable and writable throughout replication. The target filesystem is write-protected: only the replication process can write to it, while users and applications can read it. The target becomes writable only when you remove the write protection, for example, during a failover.

### Copy options

The replication policy determines how data reaches the target:

* **Full data copy**: All data and metadata are pushed from the source cluster to the target cluster as a one-way incremental copy. Use this for disaster recovery.
* **Selective copy - metadata only**: Only metadata is pushed from the source cluster to the target cluster. File data is pulled from the source cluster when accessed on the target cluster (hydration). Use this for on-demand caching.
* **Selective copy - critical paths**: Selected directory paths are pushed in full. All other data behaves as metadata-only.

With selective copy, a file whose data blocks are still on the source is in *lazy mode*: the file is visible on the target, and its data is pulled from the source when the file is accessed. To control this per file, use `weka replication fs-pair fetch` to pull a file's data to the target, and `weka replication fs-pair release` to return it to lazy mode. See [Manage file hydration on the target](https://docs.weka.io/weka-filesystems-and-object-stores/asynchronous-replication/manage-asynchronous-replication.md#manage-file-hydration-on-the-target).

### Access strategy

The access strategy determines when users see each replicated snapshot on the target:

* **Instant access** (default): The snapshot is exposed immediately, and its data is visible under the `.snapshot` directory. Data is copied in the background and retrieved on demand when accessed.
* **Copy first**: The snapshot is applied only after its data and metadata are fully copied. Use this when workloads on the target require immediate local data access with full consistency.

Copy first with a full data copy keeps all data on the target after every cycle. Its data moves together with the metadata, so its **Current Status** shows `METADATA_SYNC + COPY`, and it has no separate `COPY` phase.

## Size the target filesystem

Two constraints apply. Size for whichever is larger.

**By copy option:**

A full data copy needs a target at least the size of the source filesystem. The system checks this when you add or update the pair. If you later increase the size of the source filesystem, keep the target filesystem at least the source size, plus room for the snapshots the pair retains. To increase the size of the target filesystem, run `weka fs update <target filesystem> --total-capacity <size>` on the target cluster.

A selective copy can be far smaller. Size it for the working set, which includes the directories named in `--copy-path`, plus headroom.

A working-set target runs at a high fill level by design. When it approaches full, the system returns hydrated data to lazy mode (dehydration). Keep the working set below the dehydration threshold, or the target re-fetches data it has just released. See [Considerations](#considerations).

**By target cluster capacity:**

Size the target filesystem to at least 5% of the target cluster SSD capacity.

The same filesystem size behaves differently on different clusters. A 5 GB filesystem can work well on a small cluster but stall immediately on a large one.

{% hint style="warning" %}
If the target filesystem is smaller than about 1% of the target cluster SSD capacity, replication can stall from the first synchronization cycle, before any data is visibly transferred. Increase the filesystem size, or use a target cluster with less SSD capacity.
{% endhint %}

When the target filesystem runs out of space, the result depends on what the cycle transfers:

* **Metadata, and the data of a Copy first pair with a full data copy:** The cycle waits and retries until space is available. The pair stays in the `RUNNING` state, and its **Current Status** in `weka replication fs-pair` ends with `(stuck: Target filesystem is full)`. Free space on the target filesystem or increase its size, and the cycle continues. While the cycle runs past its interval, the system raises the replication interval alerts.
* **Data of an Instant access pair, or of a selective copy - metadata only:** Data that does not fit stays in lazy mode, and the cycle completes. To check which files are on the target, run `weka replication fs-pair hydration status`.
* **Data in a copy path of a selective copy - critical paths:** The `COPY` phase fails with `FILE_COPY_FAILED`, and the pair moves to the `ERROR` state. Free space on the target filesystem, and then resume the pair.

## Considerations

**Limits**

| Item                                 | Limit                                                                                |
| ------------------------------------ | ------------------------------------------------------------------------------------ |
| Cluster links per cluster            | 8                                                                                    |
| Replication pairs per source cluster | 8. Only the pairs that replicate from the cluster count.                             |
| Replication pairs per filesystem     | 1                                                                                    |
| Replica anchors per filesystem       | 8                                                                                    |
| Snapshots to keep per pair           | 2 to 25. Retaining more snapshots requires more storage.                             |
| Replication interval                 | 5 minutes to 30 days                                                                 |
| Copy paths per pair                  | 10. Each path up to 1,023 characters, and all paths together up to 2,048 characters. |

In a fan-in topology, where several clusters replicate to one target, the target is bounded by its 8 cluster links.

**Target filesystem and failover**

* The replication process creates the target filesystem during the first replication cycle. You cannot replicate to a filesystem that already exists.
* You can mount the target filesystem only after the first replication cycle completes.
* The target filesystem is created in the filesystem group that has the same name as the group of the source filesystem. That group must already exist on the target cluster.
* The target filesystem is write-protected while the replication pair is active. Only the replication process writes to it, and users and applications can read it. To write to it, hydrate all of its data, remove the replication pair, and then run `weka fs update <name> --access rw`.
* A target filesystem cannot be the source of another pair, so cascading replication (A to B to C) is not supported.
* Creating a manual snapshot on the target filesystem halts replication and moves the pair to the error state.
* Inode numbers on the target filesystem can differ from the source filesystem. They stay the same across replication cycles.
* Rich ACL enforcement on the source filesystem does not carry over to the target filesystem.
* A target filesystem with Data Catalog indexing enabled is not supported.
* Object stores are not supported on tiered target filesystems in a replication pair.
* Failover is manual. The system does not promote the target filesystem automatically, and it does not fail back. To return to the original source cluster, see [Fail back to the original source cluster](https://docs.weka.io/weka-filesystems-and-object-stores/asynchronous-replication/manage-asynchronous-replication.md#fail-back-to-the-original-source-cluster).
* The target cluster does not fail over automatically if its S3 endpoint becomes degraded.

**Source filesystem and data copy**

* A tiered source filesystem is supported. The target filesystem is SSD-only. For a source with 100 TB of SSD capacity and 500 TB of total capacity, set `--target-total-capacity 500TB`. The target needs 500 TB of SSD capacity.
* A source filesystem with Data Catalog indexing enabled is supported.
* Changing the policy from Selective copy - metadata only to Full data copy or Selective copy - critical paths does not copy files that were never hydrated. Hydrate those files before you change the policy.
* A selective copy can leave a file partly hydrated: part of its data is on the target, and the rest is retrieved from the source when the file is accessed.
* Dehydration on the target filesystem starts when it uses 95% of its SSD capacity and stops when usage falls to 90%. To release data outside these thresholds, run `weka replication fs-pair release`.
* The system checks the syntax of each copy path when you add or update a pair. A copy path that does not exist in the filesystem is skipped in each cycle, and the system raises a `ReplicaSnapshotCopyPathMissing` event. To correct the path, run `weka replication fs-pair update`.

**Snapshots**

* Avoid a snapshot interval shorter than 30 minutes on a filesystem that also has a replication schedule. If snapshot deletion overlaps the start of a replication cycle, the target can fall further behind than the scheduled interval.
* The **anchor snapshot** is the last live snapshot in a replication pair. It remains on the filesystem after you remove the replication pair and the cluster link, and you cannot delete it.
* Each cycle ends by deleting the snapshots beyond the `--snapshots-to-keep` limit, on the target and then on the source. By default, the cluster runs one snapshot deletion at a time across all filesystems, so this step waits for other deletions to finish. While it waits, a paused pair shows `PAUSING`. Remove the pair after the deletion finishes.

**Deployment**

* Replication is not supported on servers that run the NFS or SMB protocols, because the S3 protocol cannot be combined with NFS or SMB.
* Selective copy - critical paths requires a Data Services container on the target cluster. Set the paths in **Paths to Pre-Copy** in the GUI, or with `--copy-path` in the CLI.
* To replicate an encrypted filesystem, both clusters must use the same key management service (KMS).
* Keep the clocks of the two clusters synchronized to within 5 minutes of each other, for example with NTP.

**Related topics**

[Manage asynchronous replication using the GUI](https://docs.weka.io/weka-filesystems-and-object-stores/asynchronous-replication/manage-asynchronous-replication-using-the-gui.md)

[Manage asynchronous replication using the CLI](https://docs.weka.io/weka-filesystems-and-object-stores/asynchronous-replication/manage-asynchronous-replication.md)


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation by asking a question.

Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters:

```
GET https://docs.weka.io/weka-filesystems-and-object-stores/asynchronous-replication.md?ask=<question>&goal=<user_goal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.
