Manage object stores using the CLI

This page describes how to view and manage object stores using the CLI.

Using the CLI, you can perform the following actions:

• View object stores

• Edit an object store

• Add an object store

• View object store buckets

• Add an object store bucket

• Edit an object store bucket

• List recent operations of an object store bucket

• Delete an object store bucket

View object stores

Command: weka fs tier obs

Use this command to view information on all the object stores configured to the WEKA system.

Using the GUI, only object store buckets are present. Adding an object store bucket only adds to the present local or remote object store. If more than one is present (such as during the time recovering from a remote snapshot), use the CLI.

Edit an object store

Command: weka fs tier obs update

Use the following command line to edit an object store:

weka fs tier obs update <name> [--new-name new-name] [--site site] [--hostname=<hostname>] [--port=<port>] [--auth-method=<auth-method>] [--region=<region>] [--access-key-id=<access-key-id>] [--secret-key=<secret-key>] [--protocol=<protocol>] [--bandwidth=<bandwidth>] [--download-bandwidth=<download-bandwidth>] [--upload-bandwidth=<upload-bandwidth>] [--remove-bandwidth=<remove-bandwidth>] [--max-concurrent-downloads=<max-concurrent-downloads>] [--max-concurrent-uploads=<max-concurrent-uploads>] [--max-concurrent-removals=<max-concurrent-removals>] [--enable-upload-tags=<enable-upload-tags>]

Parameters

Parameter	Description
name *	Name of the object store to create.
new-name	New name for the object store.
site	Site location of the object store. Possible values: local - for tiering+snapshots remote - for snapshots only
hostname	Object store host identifier (hostname or IP address) to use as a default for added buckets.
port	Object store port, to be used as a default for added buckets.
auth-method	Authentication method to use as a default for added buckets. Possible values: None,AWSSignature2,AWSSignature4
region	Region name to use as a default for added buckets.
access-key-id	Object store access key ID to use as a default for added buckets.
secret-key	Object store secret key to use as a default for added buckets.
protocol	Protocol type to use as a default for added buckets. Possible values: HTTP,HTTPS,HTTPS_UNVERIFIED
bandwidth	Bandwidth limitation per core (Mbps).
download-bandwidth	Object store download bandwidth limitation per core (Mbps).
upload-bandwidth	Object store upload bandwidth limitation per core (Mbps).
remove-bandwidth	A bandwidth (Mbps) to limit the throughput of delete requests sent to the object store. Setting a bandwidth equal to or lower than the object store deletion throughput prevents an increase in the object store deletions queue.
max-concurrent-downloads	Maximum number of downloads concurrently performed on this object store in a single IO node. Possible values: 1-64
max-concurrent-uploads	Maximum number of uploads concurrently performed on this object store in a single IO node. Possible values: 1-64
max-concurrent-removals	Maximum number of removals concurrently performed on this object store in a single IO node. Possible values: 1-64
enable-upload-tags	Determines whether to enable object-tagging or not. To use as a default for added buckets. Possible values: true,false

View object store buckets

Command: weka fs tier s3

Use this command to view information on all the object store buckets configured to the WEKA system.

Add an object store bucket

Command: weka fs tier s3 add

Use the following command line to add an S3 object store:

weka fs tier s3 add <name> [--site site] [--obs-name obs-name] [--hostname=<hostname>] [--port=<port> [--bucket=<bucket>] [--auth-method=<auth-method>] [--region=<region>] [--access-key-id=<access-key-id>] [--secret-key=<secret-key>] [--protocol=<protocol>] [--bandwidth=<bandwidth>] [--download-bandwidth=<download-bandwidth>] [--remove-bandwidth=<remove-bandwidth>] [--upload-bandwidth=<upload-bandwidth>] [--errors-timeout=<errors-timeout>] [--prefetch-mib=<prefetch-mib>] [--enable-upload-tags=<enable-upload-tags>] [--max-concurrent-downloads=<max-concurrent-downloads>] [--max-concurrent-uploads=<max-concurrent-uploads>] [--max-concurrent-removals=<max-concurrent-removals>] [--max-extents-in-data-blob=<max-extents-in-data-blob>] [--max-data-blob-size=<max-data-blob-size>] [--enable-upload-tags enable-upload-tags] [--data-storage-class data-storage-class] [--metadata-storage-class metadata-storage-class][--sts-operation-type=<sts-operation-type>] [--sts-role-arn=<sts-role-arn>] [--sts-role-session-name=<sts-role-session-name>] [--sts-session-duration=<sts-session-duration>]

Parameters

Name	Description	Default
name*	Name of the object store to edit.	​
site	local - for tiering+snapshots, remote - for snapshots only. It must be the same as the object store site it is added to (obs-name).	local
obs-name	Name of the existing object store to add this object store bucket to.	If there is only one object store of type mentioned in site it is chosen automatically
hostname *	Object store host identifier or IP. Mandatory, if not specified at the object store level.	The hostname specified in obs-name if present
port	A valid object store port.	The port specified in obs-name if present, otherwise 80
bucket	A valid object store bucket name.
auth-method *	Authentication method. Possible values: None, AWSSignature2, AWSSignature4. Mandatory, if not specified in the object store level .	The auth-method specified in obs-name if present
region *	Region name. Mandatory, if not specified in the object store level .	The region specified in obs-name if present
access-key-id *	Object store bucket access key ID. Mandatory, if not specified in the object store level (can be left empty when using IAM role in AWS or GCP).	The access-key-id specified in obs-name if present
secret-key *	Object store bucket secret key. Mandatory, if not specified in the object store level (can be left empty when using IAM role in AWS or GCP).	The secret-key specified in obs-name if present
protocol	Protocol type to be used. Possible values: HTTP, HTTPS or HTTPS_UNVERIFIED.	The protocol specified in obs-name if present, otherwiseHTTP
bandwidth	Bucket bandwidth limitation per core (Mbps).
download-bandwidth	Bucket download bandwidth limitation per core (Mbps)
upload-bandwidth	Bucket upload bandwidth limitation per core (Mbps)
remove-bandwidth	A bandwidth (Mbps) to limit the throughput of delete requests sent to the object store. Setting a bandwidth equal to or lower than the object store deletion throughput prevents an increase in the object store deletions queue.
errors-timeout	If the object store link is down longer than this timeout period, all IOs that need data return an error. Possible values: 1m-15m, or 60s-900s. For example, 300s.	300s
prefetch-mib	The data size (MiB) to prefetch when reading a whole MiB on the object store.	128
enable-upload-tags	Whether to enable object-tagging or not. Possible values: true or false	false
max-concurrent-downloads	Maximum number of downloads we concurrently perform on this object store in a single IO node. Possible values: 1-64
max-concurrent-uploads	Maximum number of uploads we concurrently perform on this object store in a single IO node. Possible values: 1-64
max-concurrent-removals	Maximum number of removals we concurrently perform on this object store in a single IO node. Possible values: 1-64
max-extents-in-data-blob	Maximum number of extents' data to upload to an object store data blob.
max-data-blob-size	Maximum size to upload to an object store data blob. Format: capacity in decimal or binary units: 1B, 1KB, 1MB, 1GB, 1TB, 1PB, 1EB, 1KiB, 1MiB, 1GiB, 1TiB, 1PiB, 1EiB.
enable-upload-tags	Enable tagging of uploaded objects. For details, see object-tagging.
data-storage-class	AWS Configurable Amazon S3 storage classes, allowing users to optimize storage based on cost and access needs. Supports STANDARD, REDUCED_REDUNDANCY, STANDARD_IA, ONEZONE_IA, INTELLIGENT_TIERING, OUTPOSTS, GLACIER_IR, and EXPRESS_ONEZONE. For details, see the documentation for Amazon S3 Storage Classes. Azure Configurable Azure access storage tier, allowing users to optimize storage based on cost and access needs. Supports HOT, COOL, and COLD. For details, see the documentation for Azure Access tiers for blob data.
metadata-storage-class	AWS Configurable storage classes for the metadata on AWS. Azure Configurable Azure access storage tier for metadata. Supports HOT, COOL, and COLD.
sts-operation-type	AWS operation type to use. Possible values: assume_role or none	none
sts-role-arn	The Amazon Resource Name (ARN) of the role to assume. Mandatory when setting sts-operation to assume_role.
sts-role-session	A unique identifier for the assumed role session. The length must be between 2 and 64 characters. Allowed characters include alphanumeric characters (upper and lower case), underscore (_), equal sign (=), comma (,), period (.), at symbol (@), and hyphen (-). Space is not allowed.
sts-session-duration	The duration of the temporary security credentials in seconds. Possible values: 900 - 43200.	3600

When using the CLI, by default a misconfigured object store are not created. To create an object store even when it is misconfigured, use the --skip-verification option.

The max-concurrent settings are applied per WEKA compute process and the minimum setting of all object stores is applied.

When you create the object store bucket in AWS, to use the storage classes: S3 Intelligent-Tiering, S3 Standard-IA, S3 One Zone-IA, and S3 Glacier Instant Retrieval, do the following:

1. Create the bucket in S3 Standard.

2. Create an AWS lifecycle policy to transition objects to these storage classes.

3. Make the relevant changes and click Update to update the object store bucket.

Edit an object store bucket

Command: weka fs tier s3 update

Use the following command line to edit an object store bucket:

weka fs tier s3 update <name> [--new-name=<new-name>] [--new-obs-name new-obs-name] [--hostname=<hostname>] [--port=<port> [--bucket=<bucket>] [--auth-method=<auth-method>] [--region=<region>] [--access-key-id=<access-key-id>] [--secret-key=<secret-key>] [--protocol=<protocol>] [--bandwidth=<bandwidth>] [--download-bandwidth=<download-bandwidth>] [--upload-bandwidth=<upload-bandwidth>] [--remove-bandwidth=<remove-bandwidth>] [--errors-timeout=<errors-timeout>] [--prefetch-mib=<prefetch-mib>] [--enable-upload-tags=<enable-upload-tags>] [--max-concurrent-downloads=<max-concurrent-downloads>] [--max-concurrent-uploads=<max-concurrent-uploads>] [--max-concurrent-removals=<max-concurrent-removals>] [--max-extents-in-data-blob=<max-extents-in-data-blob>] [--max-data-blob-size=<max-data-blob-size>] [--sts-operation-type=<sts-operation-type>] [--sts-role-arn=<sts-role-arn>] [--sts-role-session-name=<sts-role-session-name>] [--sts-session-duration=<sts-session-duration>]

Parameters

Name	Value
name*	A valid name of the object store bucket to edit.
new-name	New name for the object store bucket
new-obs-name	A new object store name to add this object store bucket to. It must be an existing object store with the same site value.
hostname	Object store host identifier or IP.
port	A valid object store port
bucket	A valid object store bucket name
auth-method	Authentication method. Possible values: None, AWSSignature2 or AWSSignature4
region	Region name
access-key-id	Object store bucket access key ID
secret-key	Object store bucket secret key
protocol	Protocol type to be used. Possible values: HTTP, HTTPS or HTTPS_UNVERIFIED
bandwidth	Bandwidth limitation per core (Mbps)
download-bandwidth	Bucket download bandwidth limitation per core (Mbps)
upload-bandwidth	Bucket upload bandwidth limitation per core (Mbps)
remove-bandwidth	A bandwidth (Mbps) to limit the throughput of delete requests sent to the object store. Setting a bandwidth equal to or lower than the object store deletion throughput prevents an increase in the object store deletions queue.
errors-timeout	If the object store link is down longer than this timeout period, all IOs that need data return an error. Possible values: 1m-15m, or 60s-900s. For example, 300s.
prefetch-mib	The data size in MiB to prefetch when reading a whole MiB on the object store
enable-upload-tags	Whether to enable object-tagging or not. Possible values: true, false
max-concurrent-downloads	Maximum number of downloads we concurrently perform on this object store in a single IO node. Possible values: 1-64
max-concurrent-uploads	Maximum number of uploads we concurrently perform on this object store in a single IO node. Possible values: 1-64
max-concurrent-removals	Maximum number of removals we concurrently perform on this object store in a single IO node. Possible values: 1-64
max-extents-in-data-blob	Maximum number of extents' data to upload to an object store data blob.
max-data-blob-size	Maximum size to upload to an object store data blob. Format: capacity in decimal or binary units: 1B, 1KB, 1MB, 1GB, 1TB, 1PB, 1EB, 1KiB, 1MiB, 1GiB, 1TiB, 1PiB, 1EiB.
sts-operation-type	AWS operation type to use. Possible values: assume_role or none
sts-role-arn	The Amazon Resource Name (ARN) of the role to assume. Mandatory when setting sts-operation to assume_role.
sts-role-session	A unique identifier for the assumed role session. The length must be between 2 and 64 characters. Allowed characters include alphanumeric characters (upper and lower case), underscore (_), equal sign (=), comma (,), period (.), at symbol (@), and hyphen (-). Space is not allowed.
sts-session-duration	The duration of the temporary security credentials in seconds. Possible values: 900 - 43200.

List recent operations of an object store bucket

Command: weka fs tier ops

Use the following command line to list the recent operations running on an object store:

weka fs tier ops <name> [--format format] [--output output]...[--sort sort]...[--filter filter]...[--raw-units] [--UTC] [--no-header] [--verbose]

Parameters

Name	Value	Default
name*	A valid object store bucket name to show its recent operations.	​
format	Specify the output format. Possible values: view, csv, markdown, json, or oldview	view
output	Specify the columns in the output. Possible values: node, obsBucket, key, type, execution, phase, previous, start, size, results, errors, lastHTTP, concurrency, inode	All columns
sort	Specify the column(s) to consider when sorting the output. For the sorting order, ascending or descending, add - or + signs respectively before the column name.
filter	Specify the values to filter by in a specific column. Usage: column1=val1[,column2=val2[,..]]
raw-units	Print values in a readable format of raw units such as bytes and seconds. Possible value examples: 1KiB 234MiB 2GiB.
no-header	Don't show column headers in the output,
verbose	Show all columns in the output.

Delete an object store bucket

Command: weka fs tier s3 delete

Use the following command line to delete an object store bucket:

weka fs tier s3 delete <name>

Parameters

Name	Value
name*	A valid name of the object store bucket to delete.