W E K A
4.0
4.0
  • WEKA v4.0 documentation
  • WEKA System Overview
    • About the WEKA system
    • SSD capacity management
    • Filesystems, object stores, and filesystem groups
    • Weka networking
    • Data lifecycle management
    • Weka client and mount modes
    • Weka containers architecture overview
    • Glossary
  • Getting Started with Weka
    • Quick installation guide
    • Manage the system using the Weka CLI
    • Manage the system using the Weka GUI
    • Run first IOs with WekaFS
    • Getting started with Weka REST API
  • Planning & Installation
    • Prerequisites for installation
    • Weka installation on bare metal
      • Planning a Weka System Installation
      • Prepare the system for Weka installation
        • SR-IOV enablement
      • Obtain the Weka software installation package
      • Weka cluster installation
      • WEKA legacy system installation process
      • Add clients
    • Weka installation on AWS
      • Self-service portal
      • CloudFormation template generator
      • Deployment types
      • AWS outposts deployment
      • Supported EC2 instance types
      • Add clients
      • Auto scaling group
      • Troubleshooting
  • Performance
    • Weka performance tests
      • Test environment details
  • WekaFS Filesystems & Object Stores
    • Manage object stores
      • Manage object stores using the GUI
      • Manage object stores using the CLI
    • Manage filesystem groups
      • Manage filesystem groups using the GUI
      • Manage filesystem groups using the CLI
    • Manage filesystems
      • Manage filesystems using the GUI
      • Manage filesystems using the CLI
    • Attach or detach object store buckets
      • Attach or detach object store bucket using the GUI
      • Attach or detach object store buckets using the CLI
    • Advanced data lifecycle management
      • Advanced time-based policies for data storage location
      • Data management in tiered filesystems
      • Transition between tiered and SSD-only filesystems
      • Manual fetch and release of data
    • Mount filesystems
    • Snapshots
      • Manage snapshots using the GUI
      • Manage snapshots using the CLI
    • Snap-To-Object
      • Manage Snap-To-Object using the GUI
      • Manage Snap-To-Object using the CLI
    • Quota management
  • Additional Protocols
    • NFS
      • Manage NFS networking using the GUI
      • Manage NFS networking using the CLI
    • SMB
      • Manage SMB using the GUI
      • Manage SMB using the CLI
    • S3
      • S3 cluster management
        • Manage the S3 service using the GUI
        • Manage the S3 service using the CLI
      • S3 buckets management
        • Manage S3 buckets using the GUI
        • Manage S3 buckets using the CLI
      • S3 users and authentication
        • Manage S3 users and authentication using the CLI
        • Manage S3 service accounts using the CLI
      • S3 rules information lifecycle management (ILM)
        • Manage S3 rules using the CLI
      • Audit S3 APIs
        • Configure audit webhook using the GUI
        • Configure audit webhook using the CLI
        • Example: How to use Splunk to audit S3
      • S3 supported APIs and limitations
      • S3 examples using boto3
  • Operation Guide
    • Alerts
      • Manage alerts using the GUI
      • Manage alerts using the CLI
      • List of alerts and corrective actions
    • Events
      • Manage events using the GUI
      • Manage events using the CLI
      • List of events
    • Statistics
      • Manage statistics using the GUI
      • Manage statistics using the CLI
      • List of statistics
    • System congestion
    • Security management
      • Obtain authentication tokens
      • KMS management
        • Manage KMS using the GUI
        • Manage KMS using the CLI
      • TLS certificate management
        • Manage the TLS certificate using the GUI
        • Manage the TLS certificate using the CLI
      • CA certificate management
        • Manage the CA certificate using the GUI
        • Manage the CA certificate using the CLI
      • Account lockout threshold policy management
        • Manage the account lockout threshold policy using GUI
        • Manage the account lockout threshold policy using CLI
      • Manage the login banner
        • Manage the login banner using the GUI
        • Manage the login banner using the CLI
    • User management
      • Manage users using the GUI
      • Manage users using the CLI
    • Organizations management
      • Manage organizations using the GUI
      • Manage organizations using the CLI
      • Mount authentication for organization filesystems
    • Expand and shrink cluster resources
      • Expand and shrink overview
      • Workflow: Add a backend host
      • Expansion of specific resources
      • Shrink a Cluster
    • Background tasks
    • Upgrade Weka versions
  • Billing & Licensing
    • License overview
    • Classic license
    • Pay-As-You-Go license
  • Support
    • Prerequisites and compatibility
    • Get support for your Weka system
    • Diagnostics management
      • Traces management
        • Manage traces using the GUI
        • Manage traces using the CLI
      • Protocols debug level management
        • Manage protocols debug level using the GUI
        • Manage protocols debug level using the CLI
      • Collect and upload diagnostics data
    • Weka Home - The Weka support cloud
      • Local Weka Home overview
      • Local Weka Home deployment
      • Set the Local Weka Home to send alerts or events
      • Download the Usage Report or Analytics
  • Appendix
    • Weka CSI Plugin
    • Set up the Weka-mon external monitoring
    • Set up the SnapTool external snapshots manager
  • REST API Reference Guide
Powered by GitBook
On this page
  • Overview
  • Directory quotas
  • Working with quotas
  • Integration with the df utility
  • Manage quotas using the CLI
  • Set directory quotas/default quotas
  • List directory quotas/default quotas
  • Unsetting directory quotas/default quotas
  1. WekaFS Filesystems & Object Stores

Quota management

This page describes how to manage quotas to alert or restrict usage of the WekaFS filesystem.

PreviousManage Snap-To-Object using the CLINextNFS

Last updated 1 year ago

Overview

There are several levels on the Weka system where capacity usage can be restricted.

  1. On an organization level: Set a different organization to manage its own filesystems, where quotas for an organization can be set, as described in the section.

  2. On a filesystem level: Set a different filesystem per department/project.

  3. On a directory level: Set a different quota per project directory (useful when users are part of several projects) or per user home directory.

Directory quotas

The organization admin can set a quota on a directory. Setting a quota will start the process of counting the current directory usage. Until this process is done, the quota will not be taken into account (for empty directories, this process is instantly done).

Note: Currently, a mount point to the relevant filesystem is required to set a quota on a directory, and the quota set command should not be interrupted until the quota accounting is over.

The organization admin sets quotas to inform/restrict users from using too much of the filesystem capacity. For that, only data in the user's control is considered. Hence, the quota doesn't count the overhead of the protection bits and snapshots. It does take into account data and metadata of files in the directory, regardless if tiered or not.

Working with quotas

When working with quotas, consider the following:

  • To set a quota, the relevant filesystem must be mounted on the host where the set quota command is to be run.

  • When setting a quota, you should go through a new mount-point. This means that if you are using a host that has mounts from Weka versions before 3.10, first unmount all relevant mount point and then mount them again.

  • Quotas can be set within nested directories (up to 4 levels of nested quotas are supported) and over-provisioned under the same directory quota tree. E.g., /home can have a quota of 1TiB, and each user directory under it can have a quota of 10GiB, while there are 200 users.

  • Moving files (or directories) between two directories with quotas, into a directory with a quota, or outside a directory with a quota is not supported. The WekaFS filesystem returns EXDEV in such a case, which is usually converted by the operating system to copy and delete but is OS-dependent.

  • Quotas and hardlinks:

    • An existing hardlink is not counted as part of the quota.

    • Once a directory has a quota, it is not allowed to create a hardlink to files residing under directories with different (or without) directory quotas.

  • Restoring a filesystem from a snapshot turns the quotas back to the configuration at the time of the snapshot.

  • Creating a new filesystem from a snap-2-obj does not preserve the original quotas.

  • When working with enforcing quotas along with a writecache mount-mode, similarly to other POSIX solutions, getting above the quota might not sync all the cache writes to the backend servers. Use sync, syncfs, or fsync to commit the cached changes to the system (or fail due to exceeding the quota).

Integration with the df utility

When a hard quota is set on a directory, running the df utility will consider the hard quota as the total capacity of the directory and provide the use% relative to the quota. This can help users understand their usage and how close they are to the hard quota.

Note: The df utility behavior with quotas is currently global to the Weka system.

To change the global behavior, contact the Weka Support Team.

Manage quotas using the CLI

Set directory quotas/default quotas

Command: weka fs quota set / weka fs quota set-default

Use the following commands to set a directory quota:

weka fs quota set <path> [--soft soft] [--hard hard] [--grace grace] [--owner owner]

It is also possible to set a default quota on a directory. It does not account for this directory (or existing child directories) but will automatically set the quota on new directories created directly under it. Use the following command to set a default quota of a directory:

weka fs quota set-default <path> [--soft soft] [--hard hard] [--grace grace] [--owner owner]

Parameters

Name

Type

Value

Limitations

Mandatory

Default

path

String

Path to the directory to set the quota on.

The relevant filesystem must be mounted when setting the quota.

Yes

​

soft

Number

Soft quota limit; Exceeding this number will be shown as exceeded quota but will not be enforced until the grace period is over.

Capacity in decimal or binary units, e.g.: 1GB, 1TB, 1GiB, 1TiB, unlimited

No

unlimited

hard

Number

Hard quota limit; Exceeding this number will not allow any more writes before clearing some space in the directory.

Capacity in decimal or binary units, e.g.: 1GB, 1TB, 1GiB, 1TiB, unlimited

No

unlimited

grace

Number

Specify the grace period before the soft limit is treated as a hard limit.

Format: 1d, 1w, unlimited

No

unlimited

owner

String

An opaque string identifying the directory owner (can be a name, email, slack ID, etc.) This owner will be shown in the quota report and can be notified upon exceeding the quota.

Up to 48 characters.

No

Note: To set advisory-only quotas, use a soft quota limit without setting a grace period.

Note: When both hard and soft quotas exist, setting the value of one of them to 0 will clear this quota.

List directory quotas/default quotas

Command: weka fs quota list / weka fs quota list-default

Use the following command to list the directory quotas (by default, only exceeding quotas are listed) :

weka fs quota list [fs-name] [--snap-name snap-name] [--path path] [--under under] [--over over] [--quick] [--all]

Parameters

Name

Type

Value

Limitations

Mandatory

Default

fs-name

String

Shows quota report only on the specified filesystem.

A valid wekafs filesystem name.

No

All filesystems

snap-name

String

Shows the quota report from the time of the snapshot.

Must be a valid snapshot name and be given along with the corresponding fs-name.

No

path

String

Path to a directory. Shows quota report only on the specified directory.

The relevant filesystem must be mounted in the host running the query.

No

under

String

A path to a directory under a wekafs mount.

The relevant filesystem must be mounted in the host running the query.

No

over

Number

Shows only quotas over this percentage of usage

0-100

No

quick

Boolean

Do not resolve inode to a path (provides quicker result if the report contains many entries).

No

False

all

Boolean

Shows all the quotas, not just the exceeding ones.

No

False

Use the following command to list the directory default quotas:

weka fs quota list-default [fs-name] [--snap-name snap-name] [--path path]

Parameters

Name

Type

Value

Limitations

Mandatory

Default

fs-name

String

Shows the default quotas of the specified filesystem only.

A valid wekafs filesystem name.

No

All filesystems

snap-name

String

Shows the default quotas from the time of the snapshot.

Must be a valid snapshot name and be given along with the corresponding fs-name.

No

path

String

Path to a directory. Shows the default quotas report only on the specified directory.

The relevant filesystem must be mounted in the host running the query.

No

Unsetting directory quotas/default quotas

Command: weka fs quota unset / weka fs quota unset-default

Use the following commands to unset a directory quota:

weka fs quota unset <path>

Use the following command to unset a default quota of a directory:

weka fs quota unset-default <path>

Parameters

Name

Type

Value

Limitations

Mandatory

Default

path

String

Path to the directory to set the quota on

The relevant filesystem must be mounted when setting the quota

Yes

​

organization's usage and quota management