W E K A
4.2
4.2
  • WEKA v4.2 documentation
    • Documentation revision history
  • WEKA System Overview
    • Introduction
      • WEKA system functionality features
      • Converged WEKA system deployment
      • Optimize redundancy in WEKA deployments
    • SSD capacity management
    • Filesystems, object stores, and filesystem groups
    • WEKA networking
    • Data lifecycle management
    • WEKA client and mount modes
    • WEKA containers architecture overview
    • Glossary
  • Planning and Installation
    • Prerequisites and compatibility
    • WEKA cluster installation on bare metal servers
      • Plan the WEKA system hardware requirements
      • Obtain the WEKA installation packages
      • Install the WEKA cluster using the WMS with WSA
      • Install the WEKA cluster using the WSA
      • Manually install OS and WEKA on servers
      • Manually prepare the system for WEKA configuration
        • Broadcom adapter setup for WEKA system
        • Enable the SR-IOV
      • Configure the WEKA cluster using the WEKA Configurator
      • Manually configure the WEKA cluster using the resources generator
      • Perform post-configuration procedures
      • Add clients
    • WEKA installation on AWS
      • WEKA installation on AWS using Terraform
        • Terraform-AWS-WEKA module description
        • Deployment on AWS using Terraform
        • Required services and supported regions
        • Supported EC2 instance types using Terraform
        • WEKA cluster auto-scaling in AWS
        • Detailed deployment tutorial: WEKA on AWS using Terraform
      • WEKA installation on AWS using the Cloud Formation
        • Self-service portal
        • CloudFormation template generator
        • Deployment types
        • AWS Outposts deployment
        • Supported EC2 instance types using Cloud Formation
        • Add clients
        • Auto scaling group
        • Troubleshooting
    • WEKA installation on Azure
    • WEKA installation on GCP
      • WEKA project description
      • GCP-WEKA deployment Terraform package description
      • Deployment on GCP using Terraform
      • Required services and supported regions
      • Supported machine types and storage
      • Auto-scale instances in GCP
      • Add clients
      • Troubleshooting
  • Getting Started with WEKA
    • Manage the system using the WEKA GUI
    • Manage the system using the WEKA CLI
      • WEKA CLI hierarchy
      • CLI reference guide
    • Run first IOs with WEKA filesystem
    • Getting started with WEKA REST API
    • WEKA REST API and equivalent CLI commands
  • Performance
    • WEKA performance tests
      • Test environment details
  • WEKA Filesystems & Object Stores
    • Manage object stores
      • Manage object stores using the GUI
      • Manage object stores using the CLI
    • Manage filesystem groups
      • Manage filesystem groups using the GUI
      • Manage filesystem groups using the CLI
    • Manage filesystems
      • Manage filesystems using the GUI
      • Manage filesystems using the CLI
    • Attach or detach object store buckets
      • Attach or detach object store bucket using the GUI
      • Attach or detach object store buckets using the CLI
    • Advanced data lifecycle management
      • Advanced time-based policies for data storage location
      • Data management in tiered filesystems
      • Transition between tiered and SSD-only filesystems
      • Manual fetch and release of data
    • Mount filesystems
      • Mount filesystems from Single Client to Multiple Clusters (SCMC)
      • Manage authentication across multiple clusters with connection profiles
    • Snapshots
      • Manage snapshots using the GUI
      • Manage snapshots using the CLI
    • Snap-To-Object
      • Manage Snap-To-Object using the GUI
      • Manage Snap-To-Object using the CLI
    • Quota management
      • Manage quotas using the GUI
      • Manage quotas using the CLI
  • Additional Protocols
    • Additional protocol containers
    • Manage the NFS protocol
      • Supported NFS client mount parameters
      • Manage NFS networking using the GUI
      • Manage NFS networking using the CLI
    • Manage the S3 protocol
      • S3 cluster management
        • Manage the S3 service using the GUI
        • Manage the S3 service using the CLI
      • S3 buckets management
        • Manage S3 buckets using the GUI
        • Manage S3 buckets using the CLI
      • S3 users and authentication
        • Manage S3 users and authentication using the CLI
        • Manage S3 service accounts using the CLI
      • S3 rules information lifecycle management (ILM)
        • Manage S3 lifecycle rules using the GUI
        • Manage S3 lifecycle rules using the CLI
      • Audit S3 APIs
        • Configure audit webhook using the GUI
        • Configure audit webhook using the CLI
        • Example: How to use Splunk to audit S3
      • S3 supported APIs and limitations
      • S3 examples using boto3
    • Manage the SMB protocol
      • Manage SMB using the GUI
      • Manage SMB using the CLI
  • Operation Guide
    • Alerts
      • Manage alerts using the GUI
      • Manage alerts using the CLI
      • List of alerts and corrective actions
    • Events
      • Manage events using the GUI
      • Manage events using the CLI
      • List of events
    • Statistics
      • Manage statistics using the GUI
      • Manage statistics using the CLI
      • List of statistics
    • Insights
    • System congestion
    • Security management
      • Obtain authentication tokens
      • KMS management
        • Manage KMS using the GUI
        • Manage KMS using the CLI
      • TLS certificate management
        • Manage the TLS certificate using the GUI
        • Manage the TLS certificate using the CLI
      • CA certificate management
        • Manage the CA certificate using the GUI
        • Manage the CA certificate using the CLI
      • Account lockout threshold policy management
        • Manage the account lockout threshold policy using GUI
        • Manage the account lockout threshold policy using CLI
      • Manage the login banner
        • Manage the login banner using the GUI
        • Manage the login banner using the CLI
    • User management
      • Manage users using the GUI
      • Manage users using the CLI
    • Organizations management
      • Manage organizations using the GUI
      • Manage organizations using the CLI
      • Mount authentication for organization filesystems
    • Expand and shrink cluster resources
      • Add a backend server
      • Expand specific resources of a container
      • Shrink a cluster
    • Background tasks
      • Manage background tasks using the GUI
      • Manage background tasks using the CLI
    • Upgrade WEKA versions
  • Billing & Licensing
    • License overview
    • Classic license
  • Monitor the WEKA Cluster
    • Deploy monitoring tools using the WEKA Management Station (WMS)
    • WEKA Home - The WEKA support cloud
      • Local WEKA Home overview
      • Deploy Local WEKA Home v3.0 or higher
      • Deploy Local WEKA Home v2.x
      • Explore cluster insights and statistics
      • Manage alerts and integrations
      • Enforce security and compliance
      • Optimize support and data management
    • Set up the WEKAmon external monitoring
    • Set up the SnapTool external snapshots manager
  • Support
    • Get support for your WEKA system
    • Diagnostics management
      • Traces management
        • Manage traces using the GUI
        • Manage traces using the CLI
      • Protocols debug level management
        • Manage protocols debug level using the GUI
        • Manage protocols debug level using the CLI
      • Diagnostics data management
  • Best Practice Guides
    • WEKA and Slurm integration
      • Avoid conflicting CPU allocations
    • Storage expansion best practice
  • Appendices
    • WEKA CSI Plugin
      • Deployment
      • Storage class configurations
      • Tailor your storage class configuration with mount options
      • Dynamic and static provisioning
      • Launch an application using WEKA as the POD's storage
      • Add SELinux support
      • NFS transport failback
      • Upgrade legacy persistent volumes for capacity enforcement
      • Troubleshooting
    • Convert cluster to multi-container backend
    • Create a client image
Powered by GitBook
On this page
  • Create a bucket
  • List buckets
  • Set a bucket quota
  • Unset a bucket quota
  • Delete a bucket
  • Manage bucket policies
  • Set a pre-defined bucket policy
  • Set a custom bucket policy
  • View a bucket policy
  • Unset a bucket policy
  1. Additional Protocols
  2. Manage the S3 protocol
  3. S3 buckets management

Manage S3 buckets using the CLI

This page describes how to manage S3 buckets using the CLI.

PreviousManage S3 buckets using the GUINextS3 users and authentication

Last updated 5 months ago

Using the CLI, you can:

Create a bucket

Command: weka s3 bucket create

Use the following command line to create an S3 bucket:

weka s3 bucket create <name> [--policy policy] [--policy-json policy-json] [--hard-quota hard-quota] [--fs-name fs-name] [--fs-id fs-id] [--existing-path existing-path]

S3 does not support creating buckets on filesystems with names containing the characters ' ', '(', ')', or '&'. Verify that the filesystem name excludes these characters. Rename the filesystem if needed before creating the S3 bucket.

Parameters

Name
Value
Default

name*

policy

The name of a pre-defined bucket policy for anonymous access. Possible values: none, download, upload, public.

none

policy-json

A path to a custom policy JSON file representing an S3 bucket policy for anonymous access.

hard-quota

Hard quota for the S3 bucket. You can only set on a new bucket without existing data. You cannot set it when using existing-path to an existing directory with data.

fs-name

Existing filesystem name to create the bucket within. Possible values: fs-name, fs-id.

The default filesystem specified when creating the S3 cluster.

fs-id

Existing filesystem ID to create the bucket within. Possible values: fs-name, fs-id.

The default filesystem specified when creating the S3 cluster.

existing-path

Existing directory path relative to the filesystem root to expose a bucket from.

List buckets

Command: weka s3 bucket list

Use this command to list existing buckets.

Set a bucket quota

Command: weka s3 bucket quota set

Use the following command line to set an S3 bucket quota:

weka s3 bucket quota set <bucket-name> <hard-quota>

Parameters

Name
Value

bucket-name*

The name of an existing S3 bucket.

hard-quota*

Hard quota for the S3 bucket. You can only set it initially on an empty bucket. Calling this command on a bucket that already has a quota changes the quota limitation.

Unset a bucket quota

Command: weka s3 bucket quota unset <bucket-name>

Use this command to unset an existing bucket quota.

If the bucket point to a directory shared with other protocols, changing the quota affects all protocols (changes the associated directory quota).

Delete a bucket

Command: weka s3 bucket destroy

Use this command to delete an existing bucket from the filesystem only if the bucket is empty. If the bucket is not empty, you can keep the data on the filesystem and remove the bucket from the S3 configuration.

weka s3 bucket destroy <name> [--unlink]

Parameters

Name
Value

name*

The name of an existing S3 bucket.

unlink

Detaches the bucket from the S3 configuration and keeps the data and metadata in place. Consequently, you can recreate the bucket while preserving the data and metadata (see weka s3 bucket create using the existing-path option).

Note: If the intent is to keep the data files for use outside of the S3 configuration and delete only the S3 metadata, contact the Customer Success Team for assistance.

Manage bucket policies

It is possible to set bucket policies for anonymous access. You can choose a pre-defined policy or add a customized policy.

Set a pre-defined bucket policy

A bucket is automatically created without any anonymous access permissions. You can use one of the pre-defined policies: download, upload, or public.

Example: For a bucket named mybucket, the following are the pre-defined policy values:

{
  "Statement": [
    {
      "Action": [
        "s3:GetBucketLocation",
        "s3:ListBucket"
      ],
      "Effect": "Allow",
      "Principal": {
        "AWS": [
          "*"
        ]
      },
      "Resource": [
        "arn:aws:s3:::mybucket"
      ]
    },
    {
      "Action": [
        "s3:GetObject"
      ],
      "Effect": "Allow",
      "Principal": {
        "AWS": [
          "*"
        ]
      },
      "Resource": [
        "arn:aws:s3:::mybucket/*"
      ]
    }
  ],
  "Version": "2012-10-17"
} 
{
  "Statement": [
    {
      "Action": [
        "s3:GetBucketLocation",
        "s3:ListBucketMultipartUploads"
      ],
      "Effect": "Allow",
      "Principal": {
        "AWS": [
          "*"
        ]
      },
      "Resource": [
        "arn:aws:s3:::mybucket"
      ]
    },
    {
      "Action": [
        "s3:DeleteObject",
        "s3:ListMultipartUploadParts",
        "s3:PutObject",
        "s3:AbortMultipartUpload"
      ],
      "Effect": "Allow",
      "Principal": {
        "AWS": [
          "*"
        ]
      },
      "Resource": [
        "arn:aws:s3:::mybucket/*"
      ]
    }
  ],
  "Version": "2012-10-17"
}
{
  "Statement": [
    {
      "Action": [
        "s3:GetBucketLocation",
        "s3:ListBucket",
        "s3:ListBucketMultipartUploads"
      ],
      "Effect": "Allow",
      "Principal": {
        "AWS": [
          "*"
        ]
      },
      "Resource": [
        "arn:aws:s3:::mybucket"
      ]
    },
    {
      "Action": [
        "s3:ListMultipartUploadParts",
        "s3:PutObject",
        "s3:AbortMultipartUpload",
        "s3:DeleteObject",
        "s3:GetObject"
      ],
      "Effect": "Allow",
      "Principal": {
        "AWS": [
          "*"
        ]
      },
      "Resource": [
        "arn:aws:s3:::mybucket/*"
      ]
    }
  ],
  "Version": "2012-10-17"
}

Command: weka s3 bucket policy set

Use the following command line to set a pre-defined bucket policy:

weka s3 bucket policy set <bucket-name> <bucket-policy>

Parameters

Name
Value

bucket-name*

Name of an existing S3 bucket.

bucket-policy*

Name of a pre-defined bucket policy for anonymous access. Possible values: none, download, upload, public.

Set a custom bucket policy

For example, to set a custom policy for mybucket to allow read-only access for objects with a public/ prefix, the custom policy, as generated with the calculator, is:

{
  "Id": "Policy1624778813411",
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "Stmt1624778790840",
      "Action": [
        "s3:ListBucket"
      ],
      "Effect": "Allow",
      "Resource": "arn:aws:s3:::mybucket",
      "Condition": {
        "StringEquals": {
          "s3:prefix": "public/"
        }
      },
      "Principal": "*"
    },
    {
      "Sid": "Stmt1624778812360",
      "Action": [
        "s3:GetObject"
      ],
      "Effect": "Allow",
      "Resource": "arn:aws:s3:::mybucket/public/*",
      "Principal": "*"
    }
  ]
}

Command: weka s3 bucket policy set-custom

Use the following command line to set a custom bucket policy:

weka s3 bucket policy set-custom <bucket-name> <policy-file>

Parameters

Name
Value

bucket-name*

Name of an existing S3 bucket.

policy-file*

View a bucket policy

Command: weka s3 bucket policy get / weka s3 bucket policy get-json

Use the following command line to view an S3 bucket policy name/JSON:

weka s3 bucket policy get <bucket-name> / weka s3 bucket policy get-json <bucket-name>

Parameters

Name
Value

bucket-name*

Name of an existing S3 bucket.

Unset a bucket policy

Command: weka s3 bucket policy unset

Use the following command line to unset an S3 bucket policy:

weka s3 bucket policy unset <bucket-name>

Parameters

Name

Value

bucket-name*

Name of an existing S3 bucket.

The name for the S3 bucket to add. Refer to the section.

To create a custom policy, you can use and select S3 Bucket Policy type. With a custom policy, it is possible to limit anonymous access only to specific prefixes.

A path to a custom JSON file representing an S3 bucket policy for anonymous access. Wildcards (such as s3:*) are not allowed as an Action in the custom policy file. See .

AWS Policy Generator
Create a bucket
List buckets
Set a bucket quota
Unset a bucket quota
Delete a bucket
Manage bucket policies
Bucket Naming Limitations
Supported Policy Actions