Manage object stores using the CLI

This page describes how to view and manage object stores using the CLI.

Using the CLI, you can perform the following actions:

View object stores

Command: weka fs tier obs

Use this command to view information on all the object stores configured to the WEKA system.

Using the GUI, only object store buckets are present. Adding an object store bucket only adds to the present local or remote object store. If more than one is present (such as during the time recovering from a remote snapshot), use the CLI.

Edit an object store

Command: weka fs tier obs update

Use the following command line to edit an object store:

weka fs tier obs update <name> [--new-name new-name] [--site site] [--hostname=<hostname>] [--port=<port>] [--auth-method=<auth-method>] [--region=<region>] [--access-key-id=<access-key-id>] [--secret-key=<secret-key>] [--protocol=<protocol>] [--bandwidth=<bandwidth>] [--download-bandwidth=<download-bandwidth>] [--upload-bandwidth=<upload-bandwidth>] [--remove-bandwidth=<remove-bandwidth>] [--max-concurrent-downloads=<max-concurrent-downloads>] [--max-concurrent-uploads=<max-concurrent-uploads>] [--max-concurrent-removals=<max-concurrent-removals>] [--enable-upload-tags=<enable-upload-tags>]

Parameters

Parameter Description

Parameter	Description
`name` *	Name of the object store to create.
`new-name`	New name for the object store.
`site`	Site location of the object store. Possible values: `local` - for tiering+snapshots `remote` - for snapshots only
`hostname`	Object store host identifier (hostname or IP address) to use as a default for added buckets.
`port`	Object store port, to be used as a default for added buckets.
`auth-method`	Authentication method to use as a default for added buckets. Possible values: `None`,`AWSSignature2`,`AWSSignature4`
`region`	Region name to use as a default for added buckets.
`access-key-id`	Object store access key ID to use as a default for added buckets.
`secret-key`	Object store secret key to use as a default for added buckets.
`protocol`	Protocol type to use as a default for added buckets. Possible values: `HTTP`,`HTTPS`,`HTTPS_UNVERIFIED`
`bandwidth`	Bandwidth limitation per core (Mbps).
`download-bandwidth`	Object store download bandwidth limitation per core (Mbps).
`upload-bandwidth`	Object store upload bandwidth limitation per core (Mbps).
`remove-bandwidth`	A bandwidth (Mbps) to limit the throughput of delete requests sent to the object store. Setting a bandwidth equal to or lower than the object store deletion throughput prevents an increase in the object store deletions queue.
`max-concurrent-downloads`	Maximum number of downloads concurrently performed on this object store in a single IO node. Possible values: `1`-`64`
`max-concurrent-uploads`	Maximum number of uploads concurrently performed on this object store in a single IO node. Possible values: `1`-`64`
`max-concurrent-removals`	Maximum number of removals concurrently performed on this object store in a single IO node. Possible values: `1`-`64`
`enable-upload-tags`	Determines whether to enable object-tagging or not. To use as a default for added buckets. Possible values: `true`,`false`

name *

Name of the object store to create.

new-name

New name for the object store.

site

Site location of the object store. Possible values: local - for tiering+snapshots remote - for snapshots only

hostname

Object store host identifier (hostname or IP address) to use as a default for added buckets.

port

Object store port, to be used as a default for added buckets.

auth-method

Authentication method to use as a default for added buckets. Possible values: None,AWSSignature2,AWSSignature4

region

Region name to use as a default for added buckets.

access-key-id

Object store access key ID to use as a default for added buckets.

secret-key

Object store secret key to use as a default for added buckets.

protocol

Protocol type to use as a default for added buckets. Possible values: HTTP,HTTPS,HTTPS_UNVERIFIED

bandwidth

Bandwidth limitation per core (Mbps).

download-bandwidth

Object store download bandwidth limitation per core (Mbps).

upload-bandwidth

Object store upload bandwidth limitation per core (Mbps).

remove-bandwidth

A bandwidth (Mbps) to limit the throughput of delete requests sent to the object store. Setting a bandwidth equal to or lower than the object store deletion throughput prevents an increase in the object store deletions queue.

max-concurrent-downloads

Maximum number of downloads concurrently performed on this object store in a single IO node. Possible values: 1-64

max-concurrent-uploads

Maximum number of uploads concurrently performed on this object store in a single IO node. Possible values: 1-64

max-concurrent-removals

Maximum number of removals concurrently performed on this object store in a single IO node. Possible values: 1-64

enable-upload-tags

Determines whether to enable object-tagging or not. To use as a default for added buckets. Possible values: true,false

View object store buckets

Command: weka fs tier s3

Use this command to view information on all the object store buckets configured to the WEKA system.

Add an object store bucket

Command: weka fs tier s3 add

Use the following command line to add an S3 object store:

weka fs tier s3 add <name> [--site site] [--obs-name obs-name] [--hostname=<hostname>] [--port=<port> [--bucket=<bucket>] [--auth-method=<auth-method>] [--region=<region>] [--access-key-id=<access-key-id>] [--secret-key=<secret-key>] [--protocol=<protocol>] [--bandwidth=<bandwidth>] [--download-bandwidth=<download-bandwidth>] [--remove-bandwidth=<remove-bandwidth>] [--upload-bandwidth=<upload-bandwidth>] [--errors-timeout=<errors-timeout>] [--prefetch-mib=<prefetch-mib>] [--enable-upload-tags=<enable-upload-tags>] [--max-concurrent-downloads=<max-concurrent-downloads>] [--max-concurrent-uploads=<max-concurrent-uploads>] [--max-concurrent-removals=<max-concurrent-removals>] [--max-extents-in-data-blob=<max-extents-in-data-blob>] [--max-data-blob-size=<max-data-blob-size>] [--sts-operation-type=<sts-operation-type>] [--sts-role-arn=<sts-role-arn>] [--sts-role-session-name=<sts-role-session-name>] [--sts-session-duration=<sts-session-duration>]

Parameters

Name Description Default

Name	Description	Default
`name`*	Name of the object store to edit.
`site`	`local` - for tiering+snapshots, `remote` - for snapshots only. It must be the same as the object store site it is added to `(obs-name)`.	`local`
`obs-name`	Name of the existing object store to add this object store bucket to.	If there is only one object store of type mentioned in `site` it is chosen automatically
`hostname` *	Object store host identifier or IP. Mandatory, if not specified at the object store level.	The `hostname` specified in `obs-name` if present
`port`	A valid object store port.	The `port` specified in `obs-name` if present, otherwise 80
`bucket`	A valid object store bucket name.
`auth-method` *	Authentication method. Possible values: `None`, `AWSSignature2`, `AWSSignature4`. Mandatory, if not specified in the object store level .	The `auth-method` specified in `obs-name` if present
`region` *	Region name. Mandatory, if not specified in the object store level .	The `region` specified in `obs-name` if present
`access-key-id` *	Object store bucket access key ID. Mandatory, if not specified in the object store level (can be left empty when using IAM role in AWS or GCP).	The `access-key-id` specified in `obs-name` if present
`secret-key` *	Object store bucket secret key. Mandatory, if not specified in the object store level (can be left empty when using IAM role in AWS or GCP).	The `secret-key` specified in `obs-name` if present
`protocol`	Protocol type to be used. Possible values: `HTTP`, `HTTPS` or `HTTPS_UNVERIFIED`.	The `protocol` specified in `obs-name` if present, otherwise`HTTP`
`bandwidth`	Bucket bandwidth limitation per core (Mbps).
`download-bandwidth`	Bucket download bandwidth limitation per core (Mbps)
`upload-bandwidth`	Bucket upload bandwidth limitation per core (Mbps)
`remove-bandwidth`	A bandwidth (Mbps) to limit the throughput of delete requests sent to the object store. Setting a bandwidth equal to or lower than the object store deletion throughput prevents an increase in the object store deletions queue.
`errors-timeout`	If the object store link is down longer than this timeout period, all IOs that need data return an error. Possible values: `1m`-`15m`, or `60s`-`900s`. For example, `300s`.	`300s`
`prefetch-mib`	The data size (MiB) to prefetch when reading a whole MiB on the object store.	`128`
`enable-upload-tags`	Whether to enable object-tagging or not. Possible values: `true` or `false`	`false`
`max-concurrent-downloads`	Maximum number of downloads we concurrently perform on this object store in a single IO node. Possible values: `1`-`64`
`max-concurrent-uploads`	Maximum number of uploads we concurrently perform on this object store in a single IO node. Possible values: `1`-`64`
`max-concurrent-removals`	Maximum number of removals we concurrently perform on this object store in a single IO node. Possible values: `1`-`64`
`max-extents-in-data-blob`	Maximum number of extents' data to upload to an object store data blob.
`max-data-blob-size`	Maximum size to upload to an object store data blob. Format: capacity in decimal or binary units: 1B, 1KB, 1MB, 1GB, 1TB, 1PB, 1EB, 1KiB, 1MiB, 1GiB, 1TiB, 1PiB, 1EiB.
`sts-operation-type`	AWS operation type to use. Possible values: `assume_role` or `none`	`none`
`sts-role-arn`	The Amazon Resource Name (ARN) of the role to assume. Mandatory when setting `sts-operation` to `assume_role`.
`sts-role-session`	A unique identifier for the assumed role session. The length must be between 2 and 64 characters. Allowed characters include alphanumeric characters (upper and lower case), underscore (_), equal sign (=), comma (,), period (.), at symbol (@), and hyphen (-). Space is not allowed.
`sts-session-duration`	The duration of the temporary security credentials in seconds. Possible values: `900` - `43200`.	`3600`

name*

Name of the object store to edit.

site

local - for tiering+snapshots, remote - for snapshots only. It must be the same as the object store site it is added to (obs-name).

local

obs-name

Name of the existing object store to add this object store bucket to.

If there is only one object store of type mentioned in site it is chosen automatically

hostname *

Object store host identifier or IP. Mandatory, if not specified at the object store level.

The hostname specified in obs-name if present

port

A valid object store port.

The port specified in obs-name if present, otherwise 80

bucket

A valid object store bucket name.

auth-method *

Authentication method. Possible values: None, AWSSignature2, AWSSignature4. Mandatory, if not specified in the object store level .

The auth-method specified in obs-name if present

region *

Region name. Mandatory, if not specified in the object store level .

The region specified in obs-name if present

access-key-id *

Object store bucket access key ID. Mandatory, if not specified in the object store level (can be left empty when using IAM role in AWS or GCP).

The access-key-id specified in obs-name if present

secret-key *

Object store bucket secret key. Mandatory, if not specified in the object store level (can be left empty when using IAM role in AWS or GCP).

The secret-key specified in obs-name if present

protocol

Protocol type to be used. Possible values: HTTP, HTTPS or HTTPS_UNVERIFIED.

The protocol specified in obs-name if present, otherwiseHTTP

bandwidth

Bucket bandwidth limitation per core (Mbps).

download-bandwidth

Bucket download bandwidth limitation per core (Mbps)

upload-bandwidth

Bucket upload bandwidth limitation per core (Mbps)

remove-bandwidth

errors-timeout

If the object store link is down longer than this timeout period, all IOs that need data return an error. Possible values: 1m-15m, or 60s-900s. For example, 300s.

300s

prefetch-mib

The data size (MiB) to prefetch when reading a whole MiB on the object store.

128

enable-upload-tags

Whether to enable object-tagging or not. Possible values: true or false

false

max-concurrent-downloads

Maximum number of downloads we concurrently perform on this object store in a single IO node.

Possible values: 1-64

max-concurrent-uploads

Maximum number of uploads we concurrently perform on this object store in a single IO node.

Possible values: 1-64

max-concurrent-removals

Maximum number of removals we concurrently perform on this object store in a single IO node.

Possible values: 1-64

max-extents-in-data-blob

Maximum number of extents' data to upload to an object store data blob.

max-data-blob-size

Maximum size to upload to an object store data blob.

Format: capacity in decimal or binary units: 1B, 1KB, 1MB, 1GB, 1TB, 1PB, 1EB, 1KiB, 1MiB, 1GiB, 1TiB, 1PiB, 1EiB.

sts-operation-type

AWS operation type to use.

Possible values: assume_role or none

none

sts-role-arn

The Amazon Resource Name (ARN) of the role to assume. Mandatory when setting sts-operation to assume_role.

sts-role-session

A unique identifier for the assumed role session. The length must be between 2 and 64 characters. Allowed characters include alphanumeric characters (upper and lower case), underscore (_), equal sign (=), comma (,), period (.), at symbol (@), and hyphen (-). Space is not allowed.

sts-session-duration

The duration of the temporary security credentials in seconds.

Possible values: 900 - 43200.

3600

When using the CLI, by default a misconfigured object store are not created. To create an object store even when it is misconfigured, use the --skip-verification option.

The max-concurrent settings are applied per WEKA compute process and the minimum setting of all object stores is applied.

When you create the object store bucket in AWS, to use the storage classes: S3 Intelligent-Tiering, S3 Standard-IA, S3 One Zone-IA, and S3 Glacier Instant Retrieval, do the following:

Create the bucket in S3 Standard.
Create an AWS lifecycle policy to transition objects to these storage classes.
Make the relevant changes and click Update to update the object store bucket.

Edit an object store bucket

Command: weka fs tier s3 update

Use the following command line to edit an object store bucket:

weka fs tier s3 update <name> [--new-name=<new-name>] [--new-obs-name new-obs-name] [--hostname=<hostname>] [--port=<port> [--bucket=<bucket>] [--auth-method=<auth-method>] [--region=<region>] [--access-key-id=<access-key-id>] [--secret-key=<secret-key>] [--protocol=<protocol>] [--bandwidth=<bandwidth>] [--download-bandwidth=<download-bandwidth>] [--upload-bandwidth=<upload-bandwidth>] [--remove-bandwidth=<remove-bandwidth>] [--errors-timeout=<errors-timeout>] [--prefetch-mib=<prefetch-mib>] [--enable-upload-tags=<enable-upload-tags>] [--max-concurrent-downloads=<max-concurrent-downloads>] [--max-concurrent-uploads=<max-concurrent-uploads>] [--max-concurrent-removals=<max-concurrent-removals>] [--max-extents-in-data-blob=<max-extents-in-data-blob>] [--max-data-blob-size=<max-data-blob-size>] [--sts-operation-type=<sts-operation-type>] [--sts-role-arn=<sts-role-arn>] [--sts-role-session-name=<sts-role-session-name>] [--sts-session-duration=<sts-session-duration>]

Parameters

Name Value

Name	Value
`name`*	A valid name of the object store bucket to edit.
`new-name`	New name for the object store bucket
`new-obs-name`	A new object store name to add this object store bucket to. It must be an existing object store with the same `site` value.
`hostname`	Object store host identifier or IP.
`port`	A valid object store port
`bucket`	A valid object store bucket name
`auth-method`	Authentication method. Possible values: `None`, `AWSSignature2` or `AWSSignature4`
`region`	Region name
`access-key-id`	Object store bucket access key ID
`secret-key`	Object store bucket secret key
`protocol`	Protocol type to be used. Possible values: `HTTP`, `HTTPS` or `HTTPS_UNVERIFIED`
`bandwidth`	Bandwidth limitation per core (Mbps)
`download-bandwidth`	Bucket download bandwidth limitation per core (Mbps)
`upload-bandwidth`	Bucket upload bandwidth limitation per core (Mbps)
`remove-bandwidth`	A bandwidth (Mbps) to limit the throughput of delete requests sent to the object store. Setting a bandwidth equal to or lower than the object store deletion throughput prevents an increase in the object store deletions queue.
`errors-timeout`	If the object store link is down longer than this timeout period, all IOs that need data return an error. Possible values: `1m`-`15m`, or `60s`-`900s`. For example, `300s`.
`prefetch-mib`	The data size in MiB to prefetch when reading a whole MiB on the object store
`enable-upload-tags`	Whether to enable object-tagging or not. Possible values: `true`, `false`
`max-concurrent-downloads`	Maximum number of downloads we concurrently perform on this object store in a single IO node. Possible values: `1`-`64`
`max-concurrent-uploads`	Maximum number of uploads we concurrently perform on this object store in a single IO node. Possible values: `1`-`64`
`max-concurrent-removals`	Maximum number of removals we concurrently perform on this object store in a single IO node. Possible values: `1`-`64`
`max-extents-in-data-blob`	Maximum number of extents' data to upload to an object store data blob.
`max-data-blob-size`	Maximum size to upload to an object store data blob. Format: capacity in decimal or binary units: 1B, 1KB, 1MB, 1GB, 1TB, 1PB, 1EB, 1KiB, 1MiB, 1GiB, 1TiB, 1PiB, 1EiB.
`sts-operation-type`	AWS operation type to use. Possible values: `assume_role` or `none`
`sts-role-arn`	The Amazon Resource Name (ARN) of the role to assume. Mandatory when setting `sts-operation` to `assume_role`.
`sts-role-session`	A unique identifier for the assumed role session. The length must be between 2 and 64 characters. Allowed characters include alphanumeric characters (upper and lower case), underscore (_), equal sign (=), comma (,), period (.), at symbol (@), and hyphen (-). Space is not allowed.
`sts-session-duration`	The duration of the temporary security credentials in seconds. Possible values: `900` - `43200`.

name*

A valid name of the object store bucket to edit.

new-name

New name for the object store bucket

new-obs-name

A new object store name to add this object store bucket to. It must be an existing object store with the same site value.

hostname

Object store host identifier or IP.

port

A valid object store port

bucket

A valid object store bucket name

auth-method

Authentication method. Possible values: None, AWSSignature2 or AWSSignature4

region

Region name

access-key-id

Object store bucket access key ID

secret-key

Object store bucket secret key

protocol

Protocol type to be used. Possible values: HTTP, HTTPS or HTTPS_UNVERIFIED

bandwidth

Bandwidth limitation per core (Mbps)

download-bandwidth

Bucket download bandwidth limitation per core (Mbps)

upload-bandwidth

Bucket upload bandwidth limitation per core (Mbps)

remove-bandwidth

errors-timeout

If the object store link is down longer than this timeout period, all IOs that need data return an error. Possible values: 1m-15m, or 60s-900s. For example, 300s.

prefetch-mib

The data size in MiB to prefetch when reading a whole MiB on the object store

enable-upload-tags

Whether to enable object-tagging or not. Possible values: true, false

max-concurrent-downloads

Maximum number of downloads we concurrently perform on this object store in a single IO node.

Possible values: 1-64

max-concurrent-uploads

Maximum number of uploads we concurrently perform on this object store in a single IO node.

Possible values: 1-64

max-concurrent-removals

Maximum number of removals we concurrently perform on this object store in a single IO node.

Possible values: 1-64

max-extents-in-data-blob

Maximum number of extents' data to upload to an object store data blob.

max-data-blob-size

Maximum size to upload to an object store data blob.

Format: capacity in decimal or binary units: 1B, 1KB, 1MB, 1GB, 1TB, 1PB, 1EB, 1KiB, 1MiB, 1GiB, 1TiB, 1PiB, 1EiB.

sts-operation-type

AWS operation type to use.

Possible values: assume_role or none

sts-role-arn

The Amazon Resource Name (ARN) of the role to assume. Mandatory when setting sts-operation to assume_role.

sts-role-session

sts-session-duration

The duration of the temporary security credentials in seconds.

Possible values: 900 - 43200.

List recent operations of an object store bucket

Command: weka fs tier ops

Use the following command line to list the recent operations running on an object store:

weka fs tier ops <name> [--format format] [--output output]...[--sort sort]...[--filter filter]...[--raw-units] [--UTC] [--no-header] [--verbose]

Parameters

Name Value Default

Name	Value	Default
`name`*	A valid object store bucket name to show its recent operations.
`format`	Specify the output format. Possible values: `view`, `csv`, `markdown`, `json`, or `oldview`	`view`
`output`	Specify the columns in the output. Possible values: `node`, `obsBucket`, `key`, `type`, `execution`, `phase`, `previous`, `start`, `size`, `results`, `errors`, `lastHTTP`, `concurrency`, `inode`	All columns
`sort`	Specify the column(s) to consider when sorting the output. For the sorting order, ascending or descending, add - or + signs respectively before the column name.
`filter`	Specify the values to filter by in a specific column. Usage: `column1=val1[,column2=val2[,..]]`
`raw-units`	Print values in a readable format of raw units such as bytes and seconds. Possible value examples: `1KiB` `234MiB` `2GiB`.
`no-header`	Don't show column headers in the output,
`verbose`	Show all columns in the output.

name*

A valid object store bucket name to show its recent operations.

format

Specify the output format. Possible values: view, csv, markdown, json, or oldview

view

output

Specify the columns in the output. Possible values: node, obsBucket, key, type, execution, phase, previous, start, size, results, errors, lastHTTP, concurrency, inode

All columns

sort

Specify the column(s) to consider when sorting the output. For the sorting order, ascending or descending, add - or + signs respectively before the column name.

filter

Specify the values to filter by in a specific column. Usage: column1=val1[,column2=val2[,..]]

raw-units

Print values in a readable format of raw units such as bytes and seconds. Possible value examples: 1KiB 234MiB 2GiB.

no-header

Don't show column headers in the output,

verbose

Show all columns in the output.

Delete an object store bucket

Command: weka fs tier s3 delete

Use the following command line to delete an object store bucket:

weka fs tier s3 delete <name>

Parameters

Name Value

Name	Value
`name`*	A valid name of the object store bucket to delete.

name*

A valid name of the object store bucket to delete.

PreviousManage object stores using the GUI NextManage filesystem groups

Last updated 8 months ago