W E K A
3.14
3.14
  • WEKA v3.14 Documentation
  • Weka System Overview
    • About the WEKA System
    • SSD Capacity Management
    • Filesystems, Object Stores & Filesystem Groups
    • Weka Networking
    • Data Lifecycle Management
    • Weka Client & Mount Modes
    • Glossary
  • Getting Started with Weka
    • Quick Install Guide
    • Managing the Weka System
    • CLI Overview
    • GUI Overview
    • Serving IOs with WekaFS
  • Planning & Installation
    • Prerequisites for Installation
    • Bare Metal Installation
      • Planning a Weka System Installation
      • Setting Up the Hosts
        • SR-IOV Enablement
      • Obtaining the Weka Install File
      • Weka System Installation Process Using the CLI
      • Adding Clients
    • AWS Installation
      • Self-Service Portal
      • CloudFormation Template Generator
      • Deployment Types
      • AWS Outposts Deployment
      • Supported EC2 Instance Types
      • Adding Clients
      • Auto Scaling Group
      • Troubleshooting
  • Performance
    • Testing Weka Performance
      • Test Environment Details
  • WekaFS Filesystems
    • Managing Filesystems, Object Stores & Filesystem Groups
      • Managing Object Stores
      • Managing Filesystem Groups
      • Managing Filesystems
      • Attaching/Detaching Object Stores to/from Filesystems
      • KMS Management
    • Advanced Data Lifecycle Management
      • Advanced Time-based Policies for Data Storage Location
      • Data Management in Tiered Filesystems
      • Transition Between Tiered and SSD-Only Filesystems
      • Manual fetch and release of data
    • Mounting Filesystems
    • Snapshots
    • Snap-To-Object
    • Quota Management
  • Additional Protocols
    • NFS
    • SMB
      • SMB Management Using CLIs
      • SMB Management Using the GUI
    • S3
      • S3 Cluster Management
      • S3 Buckets Management
      • S3 Users and Authentication
      • S3 Information Lifecycle Management
      • Audit S3 APIs
      • S3 Limitations
      • S3 Examples using boto3
  • Operation Guide
    • Alerts
      • List of Alerts
    • Events
      • List of Events
    • Statistics
      • List of Statistics
    • System Congestion
    • Security
      • User Management
      • Organizations
    • Expanding & Shrinking Cluster Resources
      • Expand & Shrink Overview
      • Stages in Adding a Backend Host
      • Expansion of Specific Resources
      • Shrinking a Cluster
    • Background Tasks
    • Upgrading Weka Versions
  • Billing & Licensing
    • License Overview
    • Classic License
    • Pay-As-You-Go License
  • Support
    • Prerequisites and Compatibility
    • Getting Support for Your Weka System
    • The Weka Support Cloud
    • Diagnostics CLI Command
  • Appendix
    • Weka CSI Plugin
    • External Monitoring
    • Snapshot Management
  • REST API
Powered by GitBook
On this page
  • Considerations
  • Round Robin DNS / Load Balancer
  • S3 Service Management using the GUI
  • Creating an S3 Cluster
  • Updating an S3 Cluster Configuration
  • Deleting an S3 Cluster
  • S3 Service Management using the CLI
  • Creating an S3 Cluster
  • Checking the Status of the S3 Cluster and Hosts Readiness
  • Updating an S3 Cluster Configuration
  • Deleting an S3 Cluster
  1. Additional Protocols
  2. S3

S3 Cluster Management

This page describes how to set up, update, monitor, and delete an S3 cluster.

PreviousS3NextS3 Buckets Management

Last updated 2 years ago

Considerations

The S3 service can be exposed from the cluster hosts, ranging from three hosts to the entire cluster. The service performance scales linearly as the S3 cluster scales.

Note: Depending on the workload, you may need to use several FE cores to gain maximum performance.

Round Robin DNS / Load Balancer

To ensure that the various S3 clients will balance the load on the different Weka hosts serving S3, it is recommended to define a entry that will resolve to the list of hosts' IPs, ensuring that the clients' load will be equally distributed across all hosts. A DNS server that supports health checks can help with resiliency if any of the hosts serving S3 become unresponsive for whatever reason.

With extreme load, even a robust DNS server / load-balancer may become overloaded. You can also use a client-side load balancer, where each client checks the health of each S3 host in the cluster. One such load balancer is the open-source .

S3 Service Management using the GUI

Creating an S3 Cluster

To configure an S3 cluster, first access the S3 Service view.

To configure the S3 cluster, click the Configure button. The following Configure S3 Cluster window will be displayed:

Select which filesystem to use for the S3 service, select the Weka backend hosts to form the S3 cluster, and optionally change the port for exposing the S3 service (do not set port 9001). Then click the Configure button.

The following S3 Cluster Configuration window will be displayed:

Note: The status of the hosts will change from not ready to ready.

Updating an S3 Cluster Configuration

To update the S3 cluster, click the Update button. An Update S3 Cluster window will be displayed:

Update the list of Weka hosts or S3 service port. Then click the Update button.

Deleting an S3 Cluster

To delete a configured S3 cluster, click the Reset button in the S3 Cluster Configuration window. The following window will be displayed:

Confirm the deletion by clicking the Reset button.

S3 Service Management using the CLI

Creating an S3 Cluster

Command: weka s3 cluster create

Use the following command line to create an S3 cluster:

weka s3 cluster create <filesystem> [--all-hosts] [--host hosts] [--port port] [--anonymous-posix-uid uid] [--anonymous-posix-gid gid]

Parameters in Command Line

Name

Type

Value

Limitations

Mandatory

Default

filesystem

String

The filesystem name to be used for the S3 service

None

Yes

all-hosts

Boolean

Use all backend hosts to serve S3 commands

None

Either host list or all-hosts must be provided

Off

host

Comma-separated list of Numbers

Host IDs to serve the S3 service

Minimum of 3 hosts must be supplied.

Either host list or all-hosts must be provided

port

Number

The port where the S3 service is exposed

Do not set port 9001.

No

9000

anonymous-posix-uid

Number

POSIX UID for objects (when accessed via POSIX) created with anonymous access (for bukets with an IAM pocliy allowing that).

None

No

65534

anonymous-posix-gid

Number

POSIX GID for objects (when accessed via POSIX) created with anonymous access (for bukets with an IAM pocliy allowing that).

None

No

65534

Checking the Status of the S3 Cluster and Hosts Readiness

Command: weka s3 cluster / weka s3 cluster status

Use these commands to check the status and configuration of the S3 cluster. Once all hosts are prepared and ready, it is possible to use the S3 service.

Updating an S3 Cluster Configuration

Command: weka s3 cluster update

Use the following command line to update an S3 cluster configuration:

weka s3 cluster update [--all-hosts] [--host hosts] [--port port] [--anonymous-posix-uid uid] [--anonymous-posix-gid gid]

Parameters in Command Line

Name

Type

Value

Limitations

Mandatory

Default

all-hosts

Boolean

Use all backend hosts to serve S3 commands

None

No

host

Comma-separated list of Numbers

Host IDs to serve the S3 service

Minimum of 3 hosts

No

port

Number

The port where the S3 service is exposed

None

No

anonymous-posix-uid

Number

POSIX UID for objects (when accessed via POSIX) created with anonymous access (for bukets with an IAM pocliy allowing that).

None

No

anonymous-posix-gid

Number

POSIX GID for objects (when accessed via POSIX) created with anonymous access (for bukets with an IAM pocliy allowing that).

None

No

Deleting an S3 Cluster

Command: weka s3 cluster destroy

Use this command to destroy an S3 cluster managed by the Weka system.

Deleting an existing S3 cluster managed by the Weka system does not delete the backend Weka filesystem but removes the S3 bucket exposures of these filesystems.

Round-robin DNS
Sidekick Load Balancer
S3 Service View
Configure S3 Cluster Window
S3 Cluster Configuration Window
Update S3 Cluster Window
S3 Cluster Reset Confirmation WIndow