W E K A
4.3
4.3
  • WEKA v4.3 documentation
    • Documentation revision history
  • WEKA System Overview
    • WEKA Data Platform introduction
      • WEKA system functionality features
      • Converged WEKA system deployment
      • Optimize redundancy in WEKA deployments
    • SSD capacity management
    • Filesystems, object stores, and filesystem groups
    • WEKA networking
    • Data lifecycle management
    • WEKA client and mount modes
    • WEKA containers architecture overview
    • Glossary
  • Planning and Installation
    • Prerequisites and compatibility
    • WEKA cluster installation on bare metal servers
      • Plan the WEKA system hardware requirements
      • Obtain the WEKA installation packages
      • Install the WEKA cluster using the WMS with WSA
      • Install the WEKA cluster using the WSA
      • Manually install OS and WEKA on servers
      • Manually prepare the system for WEKA configuration
        • Broadcom adapter setup for WEKA system
        • Enable the SR-IOV
      • Configure the WEKA cluster using the WEKA Configurator
      • Manually configure the WEKA cluster using the resource generator
      • Perform post-configuration procedures
      • Add clients to an on-premises WEKA cluster
    • WEKA Cloud Deployment Manager Web (CDM Web) User Guide
    • WEKA Cloud Deployment Manager Local (CDM Local) User Guide
    • WEKA installation on AWS
      • WEKA installation on AWS using Terraform
        • Terraform-AWS-WEKA module description
        • Deployment on AWS using Terraform
        • Required services and supported regions
        • Supported EC2 instance types using Terraform
        • WEKA cluster auto-scaling in AWS
        • Detailed deployment tutorial: WEKA on AWS using Terraform
      • WEKA installation on AWS using the Cloud Formation
        • Self-service portal
        • CloudFormation template generator
        • Deployment types
        • AWS Outposts deployment
        • Supported EC2 instance types using Cloud Formation
        • Add clients to a WEKA cluster on AWS
        • Auto scaling group
        • Troubleshooting
      • Install SMB on AWS
    • WEKA installation on Azure
    • WEKA installation on GCP
      • WEKA project description
      • GCP-WEKA deployment Terraform package description
      • Deployment on GCP using Terraform
      • Required services and supported regions
      • Supported machine types and storage
      • Auto-scale instances in GCP
      • Add clients to a WEKA cluster on GCP
      • Troubleshooting
      • Detailed deployment tutorial: WEKA on GCP using Terraform
      • Google Kubernetes Engine and WEKA over POSIX deployment
  • Getting Started with WEKA
    • Manage the system using the WEKA GUI
    • Manage the system using the WEKA CLI
      • WEKA CLI hierarchy
      • CLI reference guide
    • Run first IOs with WEKA filesystem
    • Getting started with WEKA REST API
    • WEKA REST API and equivalent CLI commands
  • Performance
    • WEKA performance tests
      • Test environment details
  • WEKA Filesystems & Object Stores
    • Manage object stores
      • Manage object stores using the GUI
      • Manage object stores using the CLI
    • Manage filesystem groups
      • Manage filesystem groups using the GUI
      • Manage filesystem groups using the CLI
    • Manage filesystems
      • Manage filesystems using the GUI
      • Manage filesystems using the CLI
    • Attach or detach object store buckets
      • Attach or detach object store bucket using the GUI
      • Attach or detach object store buckets using the CLI
    • Advanced data lifecycle management
      • Advanced time-based policies for data storage location
      • Data management in tiered filesystems
      • Transition between tiered and SSD-only filesystems
      • Manual fetch and release of data
    • Mount filesystems
      • Mount filesystems from Single Client to Multiple Clusters (SCMC)
    • Snapshots
      • Manage snapshots using the GUI
      • Manage snapshots using the CLI
    • Snap-To-Object
      • Manage Snap-To-Object using the GUI
      • Manage Snap-To-Object using the CLI
    • Quota management
      • Manage quotas using the GUI
      • Manage quotas using the CLI
  • Additional Protocols
    • Additional protocol containers
    • Manage the NFS protocol
      • Supported NFS client mount parameters
      • Manage NFS networking using the GUI
      • Manage NFS networking using the CLI
    • Manage the S3 protocol
      • S3 cluster management
        • Manage the S3 service using the GUI
        • Manage the S3 service using the CLI
      • S3 buckets management
        • Manage S3 buckets using the GUI
        • Manage S3 buckets using the CLI
      • S3 users and authentication
        • Manage S3 users and authentication using the CLI
        • Manage S3 service accounts using the CLI
      • S3 rules information lifecycle management (ILM)
        • Manage S3 lifecycle rules using the GUI
        • Manage S3 lifecycle rules using the CLI
      • Audit S3 APIs
        • Configure audit webhook using the GUI
        • Configure audit webhook using the CLI
        • Example: How to use Splunk to audit S3
      • S3 supported APIs and limitations
      • S3 examples using boto3
      • Access S3 using AWS CLI
    • Manage the SMB protocol
      • Manage SMB using the GUI
      • Manage SMB using the CLI
  • Operation Guide
    • Alerts
      • Manage alerts using the GUI
      • Manage alerts using the CLI
      • List of alerts and corrective actions
    • Events
      • Manage events using the GUI
      • Manage events using the CLI
      • List of events
    • Statistics
      • Manage statistics using the GUI
      • Manage statistics using the CLI
      • List of statistics
    • Insights
    • System congestion
    • Security management
      • Obtain authentication tokens
      • KMS management
        • Manage KMS using the GUI
        • Manage KMS using the CLI
      • TLS certificate management
        • Manage the TLS certificate using the GUI
        • Manage the TLS certificate using the CLI
      • CA certificate management
        • Manage the CA certificate using the GUI
        • Manage the CA certificate using the CLI
      • Account lockout threshold policy management
        • Manage the account lockout threshold policy using GUI
        • Manage the account lockout threshold policy using CLI
      • Manage the login banner
        • Manage the login banner using the GUI
        • Manage the login banner using the CLI
      • Manage Cross-Origin Resource Sharing
    • User management
      • Manage users using the GUI
      • Manage users using the CLI
    • Organizations management
      • Manage organizations using the GUI
      • Manage organizations using the CLI
      • Mount authentication for organization filesystems
    • Expand and shrink cluster resources
      • Add a backend server
      • Expand specific resources of a container
      • Shrink a cluster
    • Background tasks
      • Set up a Data Services container for background tasks
      • Manage background tasks using the GUI
      • Manage background tasks using the CLI
    • Upgrade WEKA versions
  • Licensing
    • License overview
    • Classic license
  • Monitor the WEKA Cluster
    • Deploy monitoring tools using the WEKA Management Station (WMS)
    • WEKA Home - The WEKA support cloud
      • Local WEKA Home overview
      • Deploy Local WEKA Home v3.0 or higher
      • Deploy Local WEKA Home v2.x
      • Explore cluster insights and statistics
      • Manage alerts and integrations
      • Enforce security and compliance
      • Optimize support and data management
    • Set up the WEKAmon external monitoring
    • Set up the SnapTool external snapshots manager
  • Support
    • Get support for your WEKA system
    • Diagnostics management
      • Traces management
        • Manage traces using the GUI
        • Manage traces using the CLI
      • Protocols debug level management
        • Manage protocols debug level using the GUI
        • Manage protocols debug level using the CLI
      • Diagnostics data management
  • Best Practice Guides
    • WEKA and Slurm integration
      • Avoid conflicting CPU allocations
    • Storage expansion best practice
  • WEKApod
    • WEKApod Data Platform Appliance overview
    • WEKApod servers overview
    • Rack installation
    • WEKApod initial system setup and configuration
    • WEKApod support process
  • Appendices
    • WEKA CSI Plugin
      • Deployment
      • Storage class configurations
      • Tailor your storage class configuration with mount options
      • Dynamic and static provisioning
      • Launch an application using WEKA as the POD's storage
      • Add SELinux support
      • NFS transport failback
      • Upgrade legacy persistent volumes for capacity enforcement
      • Troubleshooting
    • Convert cluster to multi-container backend
    • Create a client image
    • Update WMS and WSA
    • BIOS tool
Powered by GitBook
On this page
  • Workflow: Local WEKA Home deployment
  • 1. Verify prerequisites
  • 2. Prepare the physical server (or VM)
  • 3. Download the Local WEKA Home bundle
  • 4. Install and configure Local WEKA Home
  • 5. Access the Local WEKA Home portal and Grafana
  • 6. Enable the WEKA cluster to send information to the Local WEKA Home
  • 7. Test the deployment
  • Upgrade the Local WEKA Home
  • Modify the Local WEKA Home configuration
  • Troubleshoot the Local WEKA Home deployment
  • Symptom: browsing to the Local WEKA Home returns an error
  • Symptom: when executing any command on the Local WEKA Home, the error “no space left” is displayed
  • Symptom: when testing the integration, the email is not received
  • Collect LWH deployment diagnostics
  1. Monitor the WEKA Cluster
  2. WEKA Home - The WEKA support cloud

Deploy Local WEKA Home v3.0 or higher

Explore procedures for deploying the Local WEKA Home v3.0 or higher, upgrading, modifying the configuration, and troubleshooting.

PreviousLocal WEKA Home overviewNextDeploy Local WEKA Home v2.x

Last updated 1 month ago

This Local WEKA Home v3.0 (or higher) runs on K3s, a lightweight Kubernetes installed on a single node cluster. Customize the deployment by specifying configuration parameters in the config.json file.

Workflow: Local WEKA Home deployment

If you have deployed the WMS, follow the procedure:Deploy monitoring tools using the WEKA Management Station (WMS). Otherwise, perform the following workflow:

1. Verify prerequisites

Verify that the following requirements are met:

  • A dedicated physical server (or VM) for the installation with systemd.

  • The user account for installing the LWH must have root privileges.

  • Server minimum CPU and RAM requirements:

    • Minimum 8 CPU cores and 20 GiB RAM for up to 1000 total processes.

      • Total processes are equal to the cores used on the cluster backends for Management/Frontend/Compute/Drives roles and the cores used on clients for Management/Frontend roles.

    • Sizing for additional processes:

      • The total number of processes determines the number of CPU and RAM required.

      • For every additional 1000 processes or less, add 1 CPU core and 8 GiB RAM.

      Example: 20 backends with 10 processes each = 200 processes; 500 clients with 2 processes each = 1000 processes. The total is 1200 processes. This deployment requires 9 CPU cores and 28 GiB.

  • SSD-backed storage requirements:

    • Minimum 500 GiB for locally collected data in /opt/wekahome/data

  • 1 Gbps network

2. Prepare the physical server (or VM)

  1. It's recommended to disable the SELinux.

  2. If enabled, it is required to disable nm-cloud-setup and reboot the node:

    systemctl disable nm-cloud-setup.service nm-cloud-setup.timer
    reboot
  3. Ensure the following ports are open and not used by any other process. Each port is used for the process specified in the brackets. homecli adds firewall rules automatically during installation for supported systems (firewalld, ufw). For any other setup, check the following ports:

    6443 (kube-apiserver)

    10259 (kube-scheduler)

    10257 (kube-controller-manager)

    10250 (kubelet)

    80 (Local WEKA Home, WEKA cluster, and web browser)

    443 (Local WEKA Home, WEKA cluster, and web browser)

  4. Ensure the following networks are trusted:

    1. 10.42.0.0/16 (pods)

    2. 10.43.0.0/16 (services)

If you forward data from the Local WEKA Home to the Cloud WEKA Home, ensure the outbound traffic on port 443 is open.

3. Download the Local WEKA Home bundle

4. Install and configure Local WEKA Home

  1. Run the Local WEKA Home setup bundle as a root user (where * is wekahome version): bash wekahome-*.bundle

  2. To update the environment, do one of the following:

    • Re-login to the server.

    • Run the following command: source /etc/profile

  3. To customize the configuration, create a config.json file from the following examples and the template located in /opt/wekahome/current/config.json.sample.

Domain

Set the domain for URL accessing the Local WEKA Home portal either by the organization domain FQDN (DNS-based) or IP address (IP-based).

The URL to access the Local WEKA Home does not accept aliases of the DNS name.

Only the name configured in the config.json or passed via CLI argument --host some.domain.com during setup can be used for accessing the Local WEKA Home.

DNS-based domain setting: In the host section at the top of the file, set the domain FQDN as shown in the following example:

{
  "host": "some.domain.com"
}

IP-based domain setting: In the host section (at the top of the file) set the IP address of the domain as shown in the following example:

{
  "host": "52.20.26.14"
}

If the host section is not set - the first IP address from the provided network interface will be used.

SMTP

To enable the Local WEKA Home to send emails, set the SMTP details in the smtp section as shown in the following example:

{
  "smtp":{
    "sender": "Weka Home",
    "host": "smtp.gmail.com",
    "port": 587,
    "user": "username@your-domain.com",
    "password": "your_password",
    "senderEmail": "weka-home-noreply@your-domain.com",
    "insecure": false,
   }
}

If your SMTP server uses a self-signed certificate, set the line "insecure": to true.

Ensure to enable the SMTP relay service in your SMTP service.

Once the Local WEKA Home is deployed, you can set it to send alerts by email, SNMP, or PagerDuty. See Manage alerts and integrations.

TLS certificates

To enforce an HTTPS connection, you can pass the TLS certificate and private key to config.json or use CLI arguments with certificate and key filenames --tls-cert cert.pem --tls-key key.pem during the next setup step.

You can generate a self-signed certificate and a private key using the following example: openssl req -x509 -newkey rsa:2048 -keyout key.pem -out cert.pem -days <days> -nodes

Example of passing TLS settings in JSON:

{
  "tls":{
    "cert":"-----BEGIN CERTIFICATE-----\n-----END CERTIFICATE-----\n",
    "key":"-----BEGIN PRIVATE KEY-----\n-----END PRIVATE KEY-----\n"
  }
}

To use an IP address as a hostname and to have a valid certificate, make sure you pass SAN during creation. A SAN or subject alternative name is a structured way to indicate all domain names and IP addresses secured by the certificate.

Related topic

TLS certificate management

Events retention period

The default number of days to keep events in the Local WEKA Home is 30. To reduce the consumption of disk space, specify the max number of days in the retentionDays section, as shown in the following example:

{
  "retentionDays": {
   "diagnostics": 10,
   "events": 20,
   "stats": 30
  }
}
Forward data from the Local WEKA Home to the Cloud WEKA Home

Set the forwarding parameters to true, as demonstrated in the sample below. Note that this is the default setting commencing from Local WEKA Home v2.10.

To turn off the forwarding of metrics from Local WEKA Home to Cloud WEKA Home, set enabled: false.

{
  "forwarding": {
    "enabled": true,
    "url": "https://api.home.weka.io",
    "enableEvents": true,
    "enableUsageReports": true,
    "enableAnalytics": true,
    "enableDiagnostics": true,
    "enableStats": true,
    "enableClusterRegistration": true
  }
}

For networks that require a proxy to forward requests, incorporate the following lines as shown in the example below:

{
  "proxy": {
    "url": "http://user:pass@server.com:3128",
    "noProxy": ["server.com"]
  }
}

The supported proxy types include:

  • http

  • https

  • socks5

Note: You don’t need to set the proxy if you are just debugging.

  1. To initialize the setup, run the following command from the root user: homecli local setup -c config.json For a fresh installation, expect approximately 5 minutes for completion. Options:

    • You can use the default configuration by running: homecli local setup

    • Specify the network interface or bind address for the cluster using: homecli local setup --iface <interface> --ip <IP address>

    • Set your domain name or external IP as the host with: homecli local setup --host <host.domain.com>

    • Enable HTTPS by providing a certificate and key directly to the command instead of using the config.json: homecli local setup --iface <interface> --tls-cert <cert.pem> --tls-key <key.pem>

Response example of a successful Local WEKA Home installation
helm status wekahome -n home-weka-io
NAME: wekahome
LAST DEPLOYED: Thu Jan  5 09:30:42 2023
NAMESPACE: home-weka-io
STATUS: deployed
REVISION: 3
TEST SUITE: None
NOTES:
Thank you for installing home-weka-io.
Your release is named homewekaio
To learn more about the release, try:

  $ helm status homewekaio -n home-weka-io
  $ helm get all homewekaio -n home-weka-io

------------------------------------------------------------------------
Weka Home Frontend:
------------------------------------------------------------------------
URL:
https://172.31.46.11
Username:
admin
To obtain password, run:
kubectl get secret -n home-weka-io weka-home-admin-credentials -o jsonpath='{.data.admin_password}' | base64 -d

------------------------------------------------------------------------
Weka Home REST API:
------------------------------------------------------------------------
URL:
https://172.31.46.11/api/

------------------------------------------------------------------------
Weka Home Statistics (Grafana):
------------------------------------------------------------------------
URL:
https://172.31.46.11/stats/
Username:
admin
To obtain password, run:
kubectl get secret -n home-weka-io weka-home-grafana-credentials  -o jsonpath='{.data.password}' | base64 -d

------------------------------------------------------------------------
Weka Home Encryption Secret Key
------------------------------------------------------------------------
To obtain secretkey, run:
kubectl get secret -n home-weka-io weka-home-encryption  -o jsonpath='{.data.encryption_secret_key}' | base64 -d

------------------------------------------------------------------------
Technical information
------------------------------------------------------------------------
Number of event store databases: 1
Easy wekahoming!

5. Access the Local WEKA Home portal and Grafana

  • URLs:

    • LWH portal: https://<your_domain>

    • Grafana: https://<your_domain>/stats/

    • WEKA Home REST API: https://<your_domain>/api/

  • Username: admin (for accessing all portals).

  • Passwords to access the URLs:

    • Obtain the LWH portal password: Run the command: kubectl get secret -n home-weka-io wekahome-admin-credentials -o jsonpath='{.data.adminPassword}' | base64 -d

    • Obtain the Grafana portal password: Run the command: kubectl get secret -n home-weka-io wekahome-grafana-credentials -o jsonpath='{.data.password}' | base64 -d

    • Obtain the WEKA Home secret key: Run the command: kubectl get secret -n home-weka-io wekahome-encryption-key -o jsonpath='{.data.encryptionKey}' | base64 -d

6. Enable the WEKA cluster to send information to the Local WEKA Home

By default, the WEKA cluster is set to send information to the public instance of WEKA Home. To get the information in the Local WEKA Home, connect to the WEKA cluster and run one of the following commands depending on the configuration:

  • Standard configuration:

weka cloud enable --cloud-url http://<ip or hostname of the Local WEKA Home server>
  • Secure configuration with valid TLS certificates:

weka cloud enable --cloud-url https://<ip or hostname of the Local WEKA Home server> 

7. Test the deployment

Access the WEKA Home portal and verify that the test data appears.

To trigger a test event, run weka events trigger-event test and verify the test event is received in the Local WEKA Home portal under the Events section.

If required, go to /var/log/pods and review the relevant log according to the timestamp (for example, wekahome-install-03-08-2023_16-29.log).

Upgrade the Local WEKA Home

The upgrade process takes up to 5 minutes. It is recommended to perform the upgrade during a maintenance window.

Upgrading from minikube or WMS to the new Local WEKA Home 3.0 bundle (based on K3s) is not supported. To upgrade, install the new Local WEKA Home bundle to the new server and add API forwarding from the minikube cluster to the new K3s cluster.

Procedure

  1. Run bash wekahome-*.bundle

  2. To modify the existing configuration, open the /opt/wekahome/config/config.json file and modify the settings. See4. Install and configure Local WEKA Home.

  3. Run homecli local upgrade. For an upgrade, it takes about 2 minutes.

  4. Run kubectl get pods -n home-weka-io and verify in the results that all pods have the status Running or Completed. To wait for the pods' statuses, run watch kubectl get pods -n home-weka-io.

  5. Verify the Local WEKA Home is upgraded successfully. Run the following command line: helm status wekahome -n home-weka-io

Modify the Local WEKA Home configuration

If there is a change in the TLS certificates, SMTP server in your environment, or any other settings in the Local WEKA Home configuration, you can modify the existing config.json with your new settings and apply them.

Procedure

  1. Open the /opt/wekahome/config/config.json file and modify the settings. See 4. Install and configure Local WEKA Home.

  2. Run homecli local upgrade

  3. Run kubectl get pods -n home-weka-io and verify in the results that all pods have the status Running or Completed. To wait for the pods' statuses, run watch kubectl get pods -n home-weka-io.

  4. Verify the Local WEKA Home is updated successfully. Run the following command line: helm status wekahome -n home-weka-io

Troubleshoot the Local WEKA Home deployment

Symptom: browsing to the Local WEKA Home returns an error

The probable cause can be, for example, a communication problem.

Resolution

  1. Check the firewall and node IP settings. If you didn't set up a firewall (see #id-2.-prepare-the-management-server), set valid rules and run:

k3s-killall.sh
homecli local upgrade
  1. Retrieve the ingress pod (controller) of the Local WEKA Home.

kubectl get pods -n kube-system -o name -l app.kubernetes.io/name=traefik
  1. Retrieve the logs and look for the error.

kubectl logs <pod name from previous command> -n kube-system > traefik.out

Symptom: when executing any command on the Local WEKA Home, the error “no space left” is displayed

The probable cause for this issue is that the containers dir (/var/lib/rancher/k3s) consumes disk space.

Resolution

Do one of the following:

  • Try to clean unused images with homecli local cleanup images

  • Resize the disk and reinstall the Local WEKA Home.

  • Relocate the K3s root directory path to a new path on a larger device (if it exists) and copy the content from the old path to the new path.

Symptom: when testing the integration, the email is not received

The probable cause can be issues related to the SMTP server, such as wrong credentials or recipient email address.

Resolution

  1. On the Integration page, select Test Integration. Wait until an error appears.

  2. Retrieve the logs and search for the error. On the Local WEKA Home terminal, run the following command:

for dep in `kubectl get deployment -n home-weka-io -o name`; do echo -----$dep-----; kubectl logs $dep -n home-weka-io --all-containers=true --timestamps=true --since=5m ; done

Collect LWH deployment diagnostics

The LWH provides a script that collects various resource details from the LWH deployed on the Kubernetes cluster and generates an archive. This information helps the Customer Success Team and R&D to analyze and provide support when troubleshooting is needed.

The LWH deployment diagnostics provide the following information:

  • Pods status

  • Logs of all pods

  • K3s settings

  • Free disk space

  • CPU and memory usage

  • Name resolutions

  • LWH version

  • Syslogs

Procedure

  1. Run the following command:

homecli local collect-debug-info [archive] [--include-sensitive] [--full-disk-scan] [--verbose]

Parameters

Parameter
Description

archive*

The path and output archive file name. For example: /path/diag/lwh_diagnostics.tar.gz

include-sensitive

Include sensitive data in the archive. For example, value overrides. Use this parameter only if required by the Customer Success Team.

full-disk-scan

Perform a higher level of disk scan. Use this parameter only if required by the Customer Success Team.

verbose

Provide a higher verbosity level of the debug information.

.

.

.

.

.

.

.

For using other operating systems, contact the .

Download the latest (v3.0 or above) to the dedicated server or VM.

The Google SMTP server requires an .

The WEKA cluster periodically and on-demand uploads data to the Local WEKA Home according to its information type (see ).

Download the latest to the dedicated physical server (or VM).

Once you generate the LWH deployment diagnostics archive file, send it to the for analysis.

Local WEKA Home bundle
app password
Local WEKA Home bundle
Customer Success Team
Verify prerequisites
Prepare the physical server (or VM)
Download the Local WEKA Home bundle
Install and configure the Local WEKA Home
Access the Local WEKA Home portal and Grafana
Enable the WEKA cluster to send data to the Local WEKA Home
Test the deployment
Which information is uploaded to WEKA Home?
Customer Success Team
Local WEKA Home v3.0 or higher deployment