Manage object stores using the CLI
This page describes how to view and manage object stores using the CLI.
Using the CLI, you can perform the following actions:
View object stores
Command: weka fs tier obs
Use this command to view information on all the object stores configured to the WEKA system.
Using the GUI, only object store buckets are present. Adding an object store bucket only adds to the present local or remote object store. If more than one is present (such as during the time recovering from a remote snapshot), use the CLI.
Edit an object store
Command: weka fs tier obs update
Use the following command line to edit an object store:
weka fs tier obs update <name> [--new-name new-name] [--site site] [--hostname=<hostname>] [--port=<port>] [--auth-method=<auth-method>] [--region=<region>] [--access-key-id=<access-key-id>] [--secret-key=<secret-key>] [--protocol=<protocol>] [--bandwidth=<bandwidth>] [--download-bandwidth=<download-bandwidth>] [--upload-bandwidth=<upload-bandwidth>] [--remove-bandwidth=<remove-bandwidth>] [--max-concurrent-downloads=<max-concurrent-downloads>] [--max-concurrent-uploads=<max-concurrent-uploads>] [--max-concurrent-removals=<max-concurrent-removals>] [--enable-upload-tags=<enable-upload-tags>]
Parameters
name *
Name of the object store to create.
new-name
New name for the object store.
site
Site location of the object store.
Possible values:
local - for tiering+snapshots
remote - for snapshots only
hostname
Object store host identifier (hostname or IP address) to use as a default for added buckets.
port
Object store port, to be used as a default for added buckets.
auth-method
Authentication method to use as a default for added buckets.
Possible values: None,AWSSignature2,AWSSignature4
region
Region name to use as a default for added buckets.
access-key-id
Object store access key ID to use as a default for added buckets.
secret-key
Object store secret key to use as a default for added buckets.
protocol
Protocol type to use as a default for added buckets.
Possible values: HTTP,HTTPS,HTTPS_UNVERIFIED
bandwidth
Bandwidth limitation per core (Mbps).
download-bandwidth
Object store download bandwidth limitation per core (Mbps).
upload-bandwidth
Object store upload bandwidth limitation per core (Mbps).
remove-bandwidth
A bandwidth (Mbps) to limit the throughput of delete requests sent to the object store. Setting a bandwidth equal to or lower than the object store deletion throughput prevents an increase in the object store deletions queue.
max-concurrent-downloads
Maximum number of downloads concurrently performed on this object store in a single IO node.
Possible values: 1-64
max-concurrent-uploads
Maximum number of uploads concurrently performed on this object store in a single IO node.
Possible values: 1-64
max-concurrent-removals
Maximum number of removals concurrently performed on this object store in a single IO node.
Possible values: 1-64
enable-upload-tags
Determines whether to enable object-tagging or not. To use as a default for added buckets.
Possible values: true,false
View object store buckets
Command: weka fs tier s3
Use this command to view information on all the object store buckets configured to the WEKA system.
Add an object store bucket
Command: weka fs tier s3 add
Use the following command line to add an S3 object store:
weka fs tier s3 add <name> [--site site] [--obs-name obs-name] [--hostname=<hostname>] [--port=<port> [--bucket=<bucket>] [--auth-method=<auth-method>] [--region=<region>] [--access-key-id=<access-key-id>] [--secret-key=<secret-key>] [--protocol=<protocol>] [--bandwidth=<bandwidth>] [--download-bandwidth=<download-bandwidth>] [--remove-bandwidth=<remove-bandwidth>] [--upload-bandwidth=<upload-bandwidth>] [--errors-timeout=<errors-timeout>] [--prefetch-mib=<prefetch-mib>] [--enable-upload-tags=<enable-upload-tags>] [--max-concurrent-downloads=<max-concurrent-downloads>] [--max-concurrent-uploads=<max-concurrent-uploads>] [--max-concurrent-removals=<max-concurrent-removals>] [--max-extents-in-data-blob=<max-extents-in-data-blob>] [--max-data-blob-size=<max-data-blob-size>] [--sts-operation-type=<sts-operation-type>] [--sts-role-arn=<sts-role-arn>] [--sts-role-session-name=<sts-role-session-name>] [--sts-session-duration=<sts-session-duration>]
Parameters
name*
Name of the object store to edit.
site
local - for tiering+snapshots,
remote - for snapshots only.
It must be the same as the object store site it is added to (obs-name).
local
obs-name
Name of the existing object store to add this object store bucket to.
If there is only one object store of type mentioned in site it is chosen automatically
hostname *
Object store host identifier or IP. Mandatory, if not specified at the object store level.
The hostname specified in obs-name if present
port
A valid object store port.
The port specified in obs-name if present, otherwise 80
bucket
A valid object store bucket name.
auth-method *
Authentication method.
Possible values: None, AWSSignature2, AWSSignature4.
Mandatory, if not specified in the object store level .
The auth-method specified in obs-name if present
region *
Region name. Mandatory, if not specified in the object store level .
The region specified in obs-name if present
access-key-id *
Object store bucket access key ID. Mandatory, if not specified in the object store level (can be left empty when using IAM role in AWS or GCP).
The access-key-id specified in obs-name if present
secret-key *
Object store bucket secret key. Mandatory, if not specified in the object store level (can be left empty when using IAM role in AWS or GCP).
The secret-key specified in obs-name if present
protocol
Protocol type to be used.
Possible values: HTTP, HTTPS or HTTPS_UNVERIFIED.
The protocol specified in obs-name if present, otherwiseHTTP
bandwidth
Bucket bandwidth limitation per core (Mbps).
download-bandwidth
Bucket download bandwidth limitation per core (Mbps)
upload-bandwidth
Bucket upload bandwidth limitation per core (Mbps)
remove-bandwidth
A bandwidth (Mbps) to limit the throughput of delete requests sent to the object store. Setting a bandwidth equal to or lower than the object store deletion throughput prevents an increase in the object store deletions queue.
errors-timeout
If the object store link is down longer than this timeout period, all IOs that need data return an error.
Possible values: 1m-15m, or 60s-900s.
For example, 300s.
300s
prefetch-mib
The data size (MiB) to prefetch when reading a whole MiB on the object store.
128
max-concurrent-downloads
Maximum number of downloads we concurrently perform on this object store in a single IO node.
Possible values: 1-64
max-concurrent-uploads
Maximum number of uploads we concurrently perform on this object store in a single IO node.
Possible values: 1-64
max-concurrent-removals
Maximum number of removals we concurrently perform on this object store in a single IO node.
Possible values: 1-64
max-extents-in-data-blob
Maximum number of extents' data to upload to an object store data blob.
max-data-blob-size
Maximum size to upload to an object store data blob.
Format: capacity in decimal or binary units: 1B, 1KB, 1MB, 1GB, 1TB, 1PB, 1EB, 1KiB, 1MiB, 1GiB, 1TiB, 1PiB, 1EiB.
sts-operation-type
AWS operation type to use.
Possible values: assume_role or none
none
sts-role-arn
The Amazon Resource Name (ARN) of the role to assume. Mandatory when setting sts-operation to assume_role.
sts-role-session
A unique identifier for the assumed role session. The length must be between 2 and 64 characters. Allowed characters include alphanumeric characters (upper and lower case), underscore (_), equal sign (=), comma (,), period (.), at symbol (@), and hyphen (-). Space is not allowed.
sts-session-duration
The duration of the temporary security credentials in seconds.
Possible values: 900 - 43200.
3600
When using the CLI, by default a misconfigured object store are not created. To create an object store even when it is misconfigured, use the --skip-verification option.
The max-concurrent settings are applied per WEKA compute process and the minimum setting of all object stores is applied.
When you create the object store bucket in AWS, to use the storage classes: S3 Intelligent-Tiering, S3 Standard-IA, S3 One Zone-IA, and S3 Glacier Instant Retrieval, do the following:
Create the bucket in S3 Standard.
Create an AWS lifecycle policy to transition objects to these storage classes.
Make the relevant changes and click Update to update the object store bucket.
Edit an object store bucket
Command: weka fs tier s3 update
Use the following command line to edit an object store bucket:
weka fs tier s3 update <name> [--new-name=<new-name>] [--new-obs-name new-obs-name] [--hostname=<hostname>] [--port=<port> [--bucket=<bucket>] [--auth-method=<auth-method>] [--region=<region>] [--access-key-id=<access-key-id>] [--secret-key=<secret-key>] [--protocol=<protocol>] [--bandwidth=<bandwidth>] [--download-bandwidth=<download-bandwidth>] [--upload-bandwidth=<upload-bandwidth>] [--remove-bandwidth=<remove-bandwidth>] [--errors-timeout=<errors-timeout>] [--prefetch-mib=<prefetch-mib>] [--enable-upload-tags=<enable-upload-tags>] [--max-concurrent-downloads=<max-concurrent-downloads>] [--max-concurrent-uploads=<max-concurrent-uploads>] [--max-concurrent-removals=<max-concurrent-removals>] [--max-extents-in-data-blob=<max-extents-in-data-blob>] [--max-data-blob-size=<max-data-blob-size>] [--sts-operation-type=<sts-operation-type>] [--sts-role-arn=<sts-role-arn>] [--sts-role-session-name=<sts-role-session-name>] [--sts-session-duration=<sts-session-duration>]
Parameters
name*
A valid name of the object store bucket to edit.
new-name
New name for the object store bucket
new-obs-name
A new object store name to add this object store bucket to. It must be an existing object store with the same site value.
hostname
Object store host identifier or IP.
port
A valid object store port
bucket
A valid object store bucket name
auth-method
Authentication method.
Possible values: None, AWSSignature2 or AWSSignature4
region
Region name
access-key-id
Object store bucket access key ID
secret-key
Object store bucket secret key
protocol
Protocol type to be used.
Possible values: HTTP, HTTPS or HTTPS_UNVERIFIED
bandwidth
Bandwidth limitation per core (Mbps)
download-bandwidth
Bucket download bandwidth limitation per core (Mbps)
upload-bandwidth
Bucket upload bandwidth limitation per core (Mbps)
remove-bandwidth
A bandwidth (Mbps) to limit the throughput of delete requests sent to the object store. Setting a bandwidth equal to or lower than the object store deletion throughput prevents an increase in the object store deletions queue.
errors-timeout
If the object store link is down longer than this timeout period, all IOs that need data return an error.
Possible values: 1m-15m, or 60s-900s.
For example, 300s.
prefetch-mib
The data size in MiB to prefetch when reading a whole MiB on the object store
enable-upload-tags
Whether to enable object-tagging or not.
Possible values: true, false
max-concurrent-downloads
Maximum number of downloads we concurrently perform on this object store in a single IO node.
Possible values: 1-64
max-concurrent-uploads
Maximum number of uploads we concurrently perform on this object store in a single IO node.
Possible values: 1-64
max-concurrent-removals
Maximum number of removals we concurrently perform on this object store in a single IO node.
Possible values: 1-64
max-extents-in-data-blob
Maximum number of extents' data to upload to an object store data blob.
max-data-blob-size
Maximum size to upload to an object store data blob.
Format: capacity in decimal or binary units: 1B, 1KB, 1MB, 1GB, 1TB, 1PB, 1EB, 1KiB, 1MiB, 1GiB, 1TiB, 1PiB, 1EiB.
sts-operation-type
AWS operation type to use.
Possible values: assume_role or none
sts-role-arn
The Amazon Resource Name (ARN) of the role to assume. Mandatory when setting sts-operation to assume_role.
sts-role-session
A unique identifier for the assumed role session. The length must be between 2 and 64 characters. Allowed characters include alphanumeric characters (upper and lower case), underscore (_), equal sign (=), comma (,), period (.), at symbol (@), and hyphen (-). Space is not allowed.
sts-session-duration
The duration of the temporary security credentials in seconds.
Possible values: 900 - 43200.
List recent operations of an object store bucket
Command: weka fs tier ops
Use the following command line to list the recent operations running on an object store:
weka fs tier ops <name> [--format format] [--output output]...[--sort sort]...[--filter filter]...[--raw-units] [--UTC] [--no-header] [--verbose]
Parameters
name*
A valid object store bucket name to show its recent operations.
format
Specify the output format.
Possible values: view, csv, markdown, json, or oldview
view
output
Specify the columns in the output.
Possible values:
node, obsBucket, key, type, execution, phase, previous, start, size, results, errors, lastHTTP, concurrency, inode
All columns
sort
Specify the column(s) to consider when sorting the output. For the sorting order, ascending or descending, add - or + signs respectively before the column name.
filter
Specify the values to filter by in a specific column. Usage: column1=val1[,column2=val2[,..]]
raw-units
Print values in a readable format of raw units such as bytes and seconds.
Possible value examples: 1KiB 234MiB 2GiB.
no-header
Don't show column headers in the output,
verbose
Show all columns in the output.
Delete an object store bucket
Command: weka fs tier s3 delete
Use the following command line to delete an object store bucket:
weka fs tier s3 delete <name>
Parameters
name*
A valid name of the object store bucket to delete.