W E K A
4.3
4.3
  • WEKA v4.3 documentation
    • Documentation revision history
  • WEKA System Overview
    • WEKA Data Platform introduction
      • WEKA system functionality features
      • Converged WEKA system deployment
      • Optimize redundancy in WEKA deployments
    • SSD capacity management
    • Filesystems, object stores, and filesystem groups
    • WEKA networking
    • Data lifecycle management
    • WEKA client and mount modes
    • WEKA containers architecture overview
    • Glossary
  • Planning and Installation
    • Prerequisites and compatibility
    • WEKA cluster installation on bare metal servers
      • Plan the WEKA system hardware requirements
      • Obtain the WEKA installation packages
      • Install the WEKA cluster using the WMS with WSA
      • Install the WEKA cluster using the WSA
      • Manually install OS and WEKA on servers
      • Manually prepare the system for WEKA configuration
        • Broadcom adapter setup for WEKA system
        • Enable the SR-IOV
      • Configure the WEKA cluster using the WEKA Configurator
      • Manually configure the WEKA cluster using the resource generator
      • Perform post-configuration procedures
      • Add clients to an on-premises WEKA cluster
    • WEKA Cloud Deployment Manager Web (CDM Web) User Guide
    • WEKA Cloud Deployment Manager Local (CDM Local) User Guide
    • WEKA installation on AWS
      • WEKA installation on AWS using Terraform
        • Terraform-AWS-WEKA module description
        • Deployment on AWS using Terraform
        • Required services and supported regions
        • Supported EC2 instance types using Terraform
        • WEKA cluster auto-scaling in AWS
        • Detailed deployment tutorial: WEKA on AWS using Terraform
      • WEKA installation on AWS using the Cloud Formation
        • Self-service portal
        • CloudFormation template generator
        • Deployment types
        • AWS Outposts deployment
        • Supported EC2 instance types using Cloud Formation
        • Add clients to a WEKA cluster on AWS
        • Auto scaling group
        • Troubleshooting
      • Install SMB on AWS
    • WEKA installation on Azure
    • WEKA installation on GCP
      • WEKA project description
      • GCP-WEKA deployment Terraform package description
      • Deployment on GCP using Terraform
      • Required services and supported regions
      • Supported machine types and storage
      • Auto-scale instances in GCP
      • Add clients to a WEKA cluster on GCP
      • Troubleshooting
      • Detailed deployment tutorial: WEKA on GCP using Terraform
      • Google Kubernetes Engine and WEKA over POSIX deployment
  • Getting Started with WEKA
    • Manage the system using the WEKA GUI
    • Manage the system using the WEKA CLI
      • WEKA CLI hierarchy
      • CLI reference guide
    • Run first IOs with WEKA filesystem
    • Getting started with WEKA REST API
    • WEKA REST API and equivalent CLI commands
  • Performance
    • WEKA performance tests
      • Test environment details
  • WEKA Filesystems & Object Stores
    • Manage object stores
      • Manage object stores using the GUI
      • Manage object stores using the CLI
    • Manage filesystem groups
      • Manage filesystem groups using the GUI
      • Manage filesystem groups using the CLI
    • Manage filesystems
      • Manage filesystems using the GUI
      • Manage filesystems using the CLI
    • Attach or detach object store buckets
      • Attach or detach object store bucket using the GUI
      • Attach or detach object store buckets using the CLI
    • Advanced data lifecycle management
      • Advanced time-based policies for data storage location
      • Data management in tiered filesystems
      • Transition between tiered and SSD-only filesystems
      • Manual fetch and release of data
    • Mount filesystems
      • Mount filesystems from Single Client to Multiple Clusters (SCMC)
    • Snapshots
      • Manage snapshots using the GUI
      • Manage snapshots using the CLI
    • Snap-To-Object
      • Manage Snap-To-Object using the GUI
      • Manage Snap-To-Object using the CLI
    • Quota management
      • Manage quotas using the GUI
      • Manage quotas using the CLI
  • Additional Protocols
    • Additional protocol containers
    • Manage the NFS protocol
      • Supported NFS client mount parameters
      • Manage NFS networking using the GUI
      • Manage NFS networking using the CLI
    • Manage the S3 protocol
      • S3 cluster management
        • Manage the S3 service using the GUI
        • Manage the S3 service using the CLI
      • S3 buckets management
        • Manage S3 buckets using the GUI
        • Manage S3 buckets using the CLI
      • S3 users and authentication
        • Manage S3 users and authentication using the CLI
        • Manage S3 service accounts using the CLI
      • S3 rules information lifecycle management (ILM)
        • Manage S3 lifecycle rules using the GUI
        • Manage S3 lifecycle rules using the CLI
      • Audit S3 APIs
        • Configure audit webhook using the GUI
        • Configure audit webhook using the CLI
        • Example: How to use Splunk to audit S3
      • S3 supported APIs and limitations
      • S3 examples using boto3
      • Access S3 using AWS CLI
    • Manage the SMB protocol
      • Manage SMB using the GUI
      • Manage SMB using the CLI
  • Operation Guide
    • Alerts
      • Manage alerts using the GUI
      • Manage alerts using the CLI
      • List of alerts and corrective actions
    • Events
      • Manage events using the GUI
      • Manage events using the CLI
      • List of events
    • Statistics
      • Manage statistics using the GUI
      • Manage statistics using the CLI
      • List of statistics
    • Insights
    • System congestion
    • Security management
      • Obtain authentication tokens
      • KMS management
        • Manage KMS using the GUI
        • Manage KMS using the CLI
      • TLS certificate management
        • Manage the TLS certificate using the GUI
        • Manage the TLS certificate using the CLI
      • CA certificate management
        • Manage the CA certificate using the GUI
        • Manage the CA certificate using the CLI
      • Account lockout threshold policy management
        • Manage the account lockout threshold policy using GUI
        • Manage the account lockout threshold policy using CLI
      • Manage the login banner
        • Manage the login banner using the GUI
        • Manage the login banner using the CLI
      • Manage Cross-Origin Resource Sharing
    • User management
      • Manage users using the GUI
      • Manage users using the CLI
    • Organizations management
      • Manage organizations using the GUI
      • Manage organizations using the CLI
      • Mount authentication for organization filesystems
    • Expand and shrink cluster resources
      • Add a backend server
      • Expand specific resources of a container
      • Shrink a cluster
    • Background tasks
      • Set up a Data Services container for background tasks
      • Manage background tasks using the GUI
      • Manage background tasks using the CLI
    • Upgrade WEKA versions
  • Licensing
    • License overview
    • Classic license
  • Monitor the WEKA Cluster
    • Deploy monitoring tools using the WEKA Management Station (WMS)
    • WEKA Home - The WEKA support cloud
      • Local WEKA Home overview
      • Deploy Local WEKA Home v3.0 or higher
      • Deploy Local WEKA Home v2.x
      • Explore cluster insights and statistics
      • Manage alerts and integrations
      • Enforce security and compliance
      • Optimize support and data management
    • Set up the WEKAmon external monitoring
    • Set up the SnapTool external snapshots manager
  • Support
    • Get support for your WEKA system
    • Diagnostics management
      • Traces management
        • Manage traces using the GUI
        • Manage traces using the CLI
      • Protocols debug level management
        • Manage protocols debug level using the GUI
        • Manage protocols debug level using the CLI
      • Diagnostics data management
  • Best Practice Guides
    • WEKA and Slurm integration
      • Avoid conflicting CPU allocations
    • Storage expansion best practice
  • WEKApod
    • WEKApod Data Platform Appliance overview
    • WEKApod servers overview
    • Rack installation
    • WEKApod initial system setup and configuration
    • WEKApod support process
  • Appendices
    • WEKA CSI Plugin
      • Deployment
      • Storage class configurations
      • Tailor your storage class configuration with mount options
      • Dynamic and static provisioning
      • Launch an application using WEKA as the POD's storage
      • Add SELinux support
      • NFS transport failback
      • Upgrade legacy persistent volumes for capacity enforcement
      • Troubleshooting
    • Convert cluster to multi-container backend
    • Create a client image
    • Update WMS and WSA
    • BIOS tool
Powered by GitBook
On this page
  • Before you begin
  • Server minimum requirements
  • SnapTool authentication
  • Option 1: Install the SnapTool package with the systemd service
  • Option 2: Install the SnapTool package in Docker
  • Edit the configuration in the snaptool.yml file
  • Snapshot naming conventions
  1. Monitor the WEKA Cluster

Set up the SnapTool external snapshots manager

The SnapTool is an external snapshots manager that enables scheduled snapshots and automatic operations

PreviousSet up the WEKAmon external monitoringNextGet support for your WEKA system

Last updated 6 months ago

WEKA provides an external snapshots manager named SnapTool, enabling scheduled snapshots for your WEKA cluster.

The SnapTool provides the following features:

  • Schedule snapshots monthly, daily, or at multiple (minute granularity) intervals during a daily schedule.

  • Set the number of snapshot copies to retain per schedule.

  • Delete expired snapshots automatically.

  • Upload snapshots to an object store automatically.

  • Upload and delete in the background.

  • Access a Web Status GUI to view the snapshot schedules, upload and download queue, , locator IDs for successfully uploaded snapshots, and logs. The default URL is http://<snaptool server hostname/IP>:8090.

The SnapTool runs on any Linux-based physical server (or VM). All communication with the WEKA cluster is done by an IP connection only to a WEKA host using the WEKA REST API.

The SnapTool package can be installed with a systemd service or Docker container. In both options, you need to edit the configuration in the snaptool.yml file before running the installation.

If you have deployed the WMS, follow the procedure in:Deploy monitoring tools using the WEKA Management Station (WMS). Otherwise, continue with this workflow.

Before you begin

If a previous SnapTool version exists in the physical server, make a copy of your existing snaptool.yml file.

If the snaptool.yml file is from releases before 1.0.0, it is incompatible with 1.0.0 or higher. You need to modify the file to use the new syntax.

Setting up a dedicated physical server (or VM) for the installation is recommended.

Server minimum requirements

  • 2 cores

  • 8 GB RAM

  • 5 GB /opt/ partition (for the SnapTool installation)

  • Network access to the WEKA cluster

  • To use Docker, the following must be installed on the dedicated physical server (or VM):

    • docker-ce

    • docker-compose or docker-compose-plugin, depending on the existing operating system.

SnapTool authentication

For the SnapTool host to communicate with the WEKA cluster, a security token is necessary. However, the SnapTool host is not required to have the WEKA client installed.

Prepare SnapTool user and token

Perform the following steps on an existing host with access to the WEKA CLI, for example, on a WEKA backend server.

  1. Create a dedicated user: Create a unique local username (for example, snaptool) for SnapTool. The unique username is displayed in the event logs, making the identification and troubleshooting of issues easier. Then, assign the ClusterAdmin or OrgAdmin role. Example: weka user add snaptool clusteradmin

  2. Generate an authentication token for the user: Run the following command: weka user login snaptool --path snaptool-authtoken.json

  3. Transfer the token: Copy the snaptool-authtoken.json file to the SnapTool management server. It will later be placed in a specific directory on that host.

  4. Remove the token file: Delete the snaptool-authtoken.json locally. Example: rm snaptool-authtoken.json

Configure SnapTool host with authentication token

Perform the following steps on the SnapTool host.

  1. Create a directory for the authentication token: Run the following command:

    mkdir /root/.weka

  2. Move the previously-created authentication token into the new directory: : Run the following command: mv ~/snaptool-authtoken.json /root/.weka/auth-token.json

  3. Ensure appropriate ownership and permissions are set: Run the following commands: chown root:root /root/.weka/auth-token.json chmod 400 /root/.weka/auth-token.json

Related topics

Obtain authentication tokens

Option 1: Install the SnapTool package with the systemd service

  1. Install the unit file into the systemd and start the service. Run the following command: ./install.sh The installer validates the connection to the cluster by the hosts specified in the snaptool.yml file.

If the systemd service is already running locally, the installer stops it and preserves the existing snaptool.yml file before restarting it.

Option 2: Install the SnapTool package in Docker

The snaptool container runs similarly to other WEKA Docker containers.

  1. Download the docker image from the docker hub. Run the following command: docker pull wekasolutions/snaptool:latest

    • snaptool.yml

    • docker_run.sh

  2. Edit the time_zone field in the docker_run.sh file.

  3. Run the following command: ./docker_run.sh

  4. Verify that the SnapTool container is running using the following command: docker ps

Example:

oot@weka142:~# docker ps
CONTAINER ID   IMAGE                   COMMAND                 CREATED      STATUS     PORTS   NAMES
718486e75b38   wekasolutions/snaptool  "/wekabin/snaptool -…"  30 hours ago Up 5 hours         weka_snaptool

A logs directory is created in the current working directory for logs and snapshot journaling files.

Edit the configuration in the snaptool.yml file

The SnapTool configuration is defined in the snaptool.yml file.

  1. Go to the snaptool directory and open the snaptool.yml file.

  2. In the cluster section under the hosts list, replace the hostnames with the actual hostnames/IP addresses of the Weka containers (up to three would be sufficient).

Syntax:

cluster:
    auth_token_file: auth-token.json
    hosts: vweka01,vweka02,vweka03

Example:

cluster:
    auth_token_file: auth-token.json
    hosts: hostname1,hostname2,hostname3

3. In the snaptool section, the default network port to access the Web Status GUI is 8090. If required, you can modify it. To disable the Web Status GUI, set the port to 0.

Syntax:

snaptool:
    port: 8090

4. In the filesystems section, specify the filesystems and their schedule names to run snapshots.

Syntax:

<fs_name1>:  <schedule1>,<schedule2>...
<fs_name2>:  <schedule1>,<schedule2>...

Example:

filesystems:
   fs01: default
   fs02: Weekdays-6pm, Weekends-noon

5. Optional. Customize the snapshot schedules.

Adhere to the following rules when customizing the schedules:

  • Schedules within a schedules group, such as default, cannot be assigned separately from the group. Use only the group name.

  • To set a specific schedule within a schedules group, such as monthly and weekly, not to run on a filesystem, remove it from the filesystem's schedule list.

  • When deleting snapshots automatically, based on the retain: value, snapshots for a schedule and filesystem are sorted by the creation time. The oldest snapshots are deleted until the number of snapshots to retain (the value specified in the retain: section) remains.

  • The SnapTool checks if the snaptool.yml file has changed about every minute and reloads it if it is changed. Snapshot schedules are then recalculated before creating new snapshots.

For details about the syntax of the schedules section, see the comments in the snaptool.yml file.

Example:

schedules:
   default:
      monthly:
         every: month
         retain: 6
         # day: 1   (this is default)
         # at: 0000 (this is default)
      weekly:
         every: Sunday
         retain: 8
         # at: 0000 (this is default)
      daily:
         every: Mon,Tue,Wed,Thu,Fri,Sat
         retain: 14
         # at: 0000 (this is default)
      hourly:
         every: Mon,Tue,Wed,Thu,Fri
         retain: 8
         interval: 60
         at: 9:00am
         until: 5pm
   workhoursHourlyUp:
      every: Mon,Tue,Wed,Thu,Fri
      retain: 7
      at: 0900
      until: 5pm
      interval: 60
      upload: True
   workhoursEvery20:
      every: Mon,Tue,Wed,Thu,Fri
      retain: 7
      at: 0900
      until: 5pm
      interval: 20
   weekendsNoon:
      every: Sat,Sun
      retain: 4
      at: 1200
   fridayUpload:
      every: Friday
      retain: 3
      at: 7pm
      upload: True

Snapshot naming conventions

The format of the snapshot names is <schedulename>.YYMMDDHHMM, with the access point @GMT-YYYY.MM.DD-HH.MM.SS.

Example: For a snapshot name Weekends-noon.2103101200 and access point @GMT-2021.03.10-12.00.00, the snapshot name is in the local timezone, the access point is in GMT, and the server timezone is GMT.

The name for a group of snapshots is<schedulegroupname>_<schedulename>.YYMMDDHHMM. The length of the full name before the '.' is a maximum of 18 characters.

Example: The default schedule group with an hourly schedule can be named default_hourly.YYMMDDHHMM.

The SnapTool distinguishes between user-created snapshots and scheduled snapshots only by their name.

When creating user-created snapshots, avoid name collisions with scheduled snapshot names. The SnapTool might automatically select the user-created snapshots for deletion if the same naming format is used.

Related topic

Snapshots

For instructions on the Docker installation, see the .

Download the latest snaptool.tar file from this and extract it to the physical server.

Edit the snaptool.yml configuration file (default location: /opt/weka/snaptool). See . This is a mandatory step before running the installer. Otherwise, the installation fails.

Download the following files from GitHub to a dedicated directory in the physical server:

Edit the snaptool.yml configuration file (default location: /opt/weka/snaptool). See . This is a mandatory step before running the installer. Otherwise, the installation fails.

Docker website
link
https://github.com/weka/snaptool/releases
Edit the configuration in snaptool.yml
Edit the configuration in snaptool.yml
Create a local user
SnapTool setup