search

Deploy Local WEKA Home on K3s

Manage the deployment, upgrade, and maintenance of Local WEKA Home (LWH) on K3s. This single-node architecture provides an on-premises observability solution for WEKA clusters.

The LWH deployment provides an on-premises observability and monitoring solution for WEKA clusters. Organizations use this model to operate within their own infrastructure instead of relying on the WEKA-hosted cloud service. Running on K3s offers a simplified, lightweight Kubernetes environment that provides control over system resources and data.

Deploying LWH on K3s supports single-node cluster configurations optimized for small to medium environments. This architecture utilizes a streamlined Kubernetes distribution to provide automated recovery and simplified lifecycle management of the LWH components.

circle-info

Deployment on K3s is supported for LWH versions 3.x, 4.x, and above.

The deployment is managed through a configuration file to ensure a consistent, reproducible, and upgradeable installation process.

Local WEKA Home deployment on K3s

hashtag
Workflow: Local WEKA Home deployment

If you have deployed the WMS and do not require IPv6 networking, follow the procedure:Deploy monitoring tools using the WEKA Management Station (WMS). Otherwise, perform the following workflow:

  1. Verify prerequisites.

  2. Prepare the physical server (or VM).

  3. Configure IPv6 (optional).

  4. Download the Local WEKA Home bundle.

  5. Install and configure the Local WEKA Home.

  6. Access the Local WEKA Home portal and Grafana.

  7. Enable the WEKA cluster to send data to the Local WEKA Home.

  8. Test the deployment.

hashtag
1. Verify prerequisites

Verify that the following requirements are met:

  • A dedicated physical server (or VM) for the installation with systemd.

  • The user account for installing the LWH must have root privileges.

  • Server minimum CPU core and RAM requirements:

    • Minimum 8 CPU cores and 20 GiB RAM for up to 1000 total processes.

      • Total processes are equal to the cores used on the cluster backends for Management/Frontend/Compute/Drives roles and the cores used on clients for Management/Frontend roles.

    • Sizing for additional processes:

      • The total number of processes determines the number of CPU cores and RAM required.

      • For every additional 1000 processes or less, add 1 CPU core and 8 GiB RAM.

      Example: 20 backends with 10 processes each = 200 processes; 500 clients with 2 processes each = 1000 processes. The total is 1200 processes. This deployment requires 9 CPU cores and 28 GiB.

  • SSD-backed storage requirements:

    • Minimum 500 GiB for locally collected data in /opt/wekahome/data

  • 1 Gbps network

circle-check

For using other operating systems, contact the Customer Success Team.

hashtag
2. Prepare the physical server (or VM)

  1. If you added an extra disk to hold the /opt/wekahome data, make sure to format and mount it. Then ensure it is remounted on reboot.

  2. It's recommended to disable the SELinux.

  3. If enabled, it is required to disable nm-cloud-setup and reboot the node:

  4. Ensure the following ports are open and not used by any other process. Each port is used for the process specified in the brackets. homecli adds firewall rules automatically during installation for supported systems (firewalld, ufw). For any other setup, check the following ports:

    6443 (kube-apiserver)

    10259 (kube-scheduler)

    10257 (kube-controller-manager)

    10250 (kubelet)

    80 (Local WEKA Home, WEKA cluster, and web browser)

    443 (Local WEKA Home, WEKA cluster, and web browser)

  5. Ensure the following networks are trusted:

    1. 10.42.0.0/16 (pods)

    2. 10.43.0.0/16 (service)

    If these networks are not available, you can customize them in the configuration file before running the installation.

circle-info

If you forward data from the Local WEKA Home to the Cloud WEKA Home, ensure the outbound traffic on port 443 is open.

hashtag
3. Configure IPv6 (optional)

Local WEKA Home (LWH) supports dual-stack networking, allowing communication over both IPv4 and IPv6 protocols. Enabling IPv6 ensures LWH can function in modern network environments that require IPv6 compliance and connectivity. Both IPv4 and IPv6 can operate simultaneously, providing flexibility and compatibility.

Benefits of IPv6 configuration

  • Support IPv6-only and dual-stack environments for seamless communication.

  • Enable network segmentation to meet enterprise deployment requirements.

  • Ensure compatibility with IPv6-enabled client applications.

  • Future-proof LWH deployments for evolving IPv6 adoption.

Prerequisites

Before configuring IPv6 support, ensure the following requirements are met:

  • The host system has IPv6 networking enabled.

  • Client machines have IPv6 connectivity to access LWH over IPv6.

  • Required IPv6 network ranges are available. If unavailable, customize them (refer to the relevant step in the procedure):

    • Pod network: 2001:cafe:42::/56

    • Service network: 2001:cafe:43::/112

  • LWH installation has not been completed.

circle-info

Dual-stack networking must be configured during cluster creation. It cannot be enabled later if the cluster was initially deployed with IPv4-only.

Procedure

  1. Verify IPv6 readiness:

Ensure the output is 0 to confirm IPv6 is enabled.

  1. Verify network interface IPv6 assignment:

Confirm the presence of a global IPv6 address (scopeid 0x0<global>) on your intended network interface.

Example:

  1. If using custom network ranges for dual-stack networking, modify the k3s arguments in your LWH configuration:

Replace the cluster-cidr and service-cidr values with your available networks.

  1. Validate the IPv6 configuration after the LWH deployment (see IPv6 validation).

hashtag
4. Download the Local WEKA Home bundle

Download the latest Local WEKA Home bundlearrow-up-right (v3.0 or above) to the dedicated server or VM.

hashtag
5. Install and configure Local WEKA Home

  1. Run the Local WEKA Home setup bundle as a root user (where * is wekahome version): bash wekahome-*.bundle

  2. To customize the configuration, create a config.json file from the following examples and the template located in /opt/wekahome/current/config.json.sample.

chevron-rightGitHub SSO integrationhashtag

Centralize and secure organizational access through GitHub Single Sign-On (SSO), simplifying authentication and user management.

Prerequisites: Prepare GitHub OAuth App

  1. Go to GitHub Developer Settings.

  2. Create a new OAuth application.

  3. Set the Authorization Callback URL the same as your Homepage URL.

  4. Note the Client ID and Client Secret.

Implement GitHub-based login

Add the following lines to the configuration file.

chevron-rightTrusted network for pods and services (optional)hashtag

If the networks for the pods (cluster) and service (10.42.0.0/16 and 10.43.0.0/16, are not available, set your available networks as shown in the following example:

Replace the cluster-cidr and service-cidr values with your available networks.

chevron-rightDomainhashtag

Set the domain for URL accessing the Local WEKA Home portal either by the organization domain FQDN (DNS-based) or IP address (IP-based).

The URL to access the Local WEKA Home does not accept aliases of the DNS name.

Only the name configured in the config.json or passed via CLI argument --host some.domain.com during setup can be used for accessing the Local WEKA Home.

DNS-based domain setting: In the host section at the top of the file, set the domain FQDN as shown in the following example:

IP-based domain setting: In the host section (at the top of the file) set the IP address of the domain as shown in the following example:

If the host section is not set - the first IP address from the provided network interface will be used.

chevron-rightSMTPhashtag

To enable the Local WEKA Home to send emails, set the SMTP details in the smtp section as shown in the following example:

If your SMTP server uses a self-signed certificate, set the line "insecure": to true.

Ensure to enable the SMTP relay service in your SMTP service.

The Google SMTP server requires an app passwordarrow-up-right.

Once the Local WEKA Home is deployed, you can set it to send alerts by email, SNMP, or PagerDuty. See Manage alerts and integrations.

chevron-rightTLS certificateshashtag

To enforce an HTTPS connection, you can pass the TLS certificate and private key to config.json or use CLI arguments with certificate and key filenames --tls-cert cert.pem --tls-key key.pem during the next setup step.

You can generate a self-signed certificate and a private key using the following example: openssl req -x509 -newkey rsa:2048 -keyout key.pem -out cert.pem -days <days> -nodes

Example of passing TLS settings in JSON:

To use an IP address as a hostname and to have a valid certificate, make sure you pass SAN during creation. A SAN or subject alternative name is a structured way to indicate all domain names and IP addresses secured by the certificate.

Related topic

Manage TLS certificates

chevron-rightEvents retention periodhashtag

The default number of days to keep events in the Local WEKA Home is 30. To reduce the consumption of disk space, specify the max number of days in the retentionDays section, as shown in the following example:

chevron-rightForward data from the Local WEKA Home to the Cloud WEKA Homehashtag

Set the forwarding parameters to true, as demonstrated in the sample below. Note that this is the default setting commencing from Local WEKA Home v2.10.

To turn off the forwarding of metrics from Local WEKA Home to Cloud WEKA Home, set enabled: false.

For networks that require a proxy to forward requests, incorporate the following lines as shown in the example below:

The supported proxy types include:

  • http

  • https

  • socks5

Note: You don’t need to set the proxy if you are just debugging.

  1. To reload your path variable do one of the following:

    • Re-login to the server.

    • Run the following command: source /etc/profile

  2. To initialize the setup, run the following command from the root user: homecli local setup -c config.json For a fresh installation, expect approximately 5 minutes for completion.

    circle-check

    If you get a "command not found" error, make sure you did not skip step 4 above.

    Options:

    • You can use the default configuration by running: homecli local setup

    • Specify the network interface or bind address for the cluster using: homecli local setup --iface <interface> --ip <IP address>

    • Set your domain name or external IP as the host with: homecli local setup --host <host.domain.com>

    • Enable HTTPS by providing a certificate and key directly to the command instead of using the config.json: homecli local setup --iface <interface> --tls-cert <cert.pem> --tls-key <key.pem>

chevron-rightResponse example of a successful Local WEKA Home installationhashtag

hashtag
6. Access the Local WEKA Home portal and Grafana

  • URLs:

    • LWH portal: https://<your_domain>

    • Grafana: https://<your_domain>/stats/

    • WEKA Home REST API: https://<your_domain>/api/

  • Username: admin (for accessing all portals).

  • Passwords to access the URLs:

    • Obtain the LWH portal password: Run the command: kubectl get secret -n home-weka-io wekahome-admin-credentials -o jsonpath='{.data.adminPassword}' | base64 -d

    • Obtain the Grafana portal password: Run the command: kubectl get secret -n home-weka-io wekahome-grafana-credentials -o jsonpath='{.data.password}' | base64 -d

    • Obtain the WEKA Home secret key: Run the command: kubectl get secret -n home-weka-io wekahome-encryption-key -o jsonpath='{.data.encryptionKey}' | base64 -d

hashtag
7. Enable the WEKA cluster to send information to the Local WEKA Home

By default, the WEKA cluster is set to send information to the public instance of WEKA Home. To get the information in the Local WEKA Home, connect to the WEKA cluster and run one of the following commands depending on the configuration:

  • Standard configuration:

  • Secure configuration with valid TLS certificates:

hashtag
8. Test the deployment

The WEKA cluster periodically and on-demand uploads data to the Local WEKA Home according to its information type (see Which information is uploaded to WEKA Home?).

  1. Access the WEKA Home portal and verify that the test data appears.

  2. To trigger a test event, run weka events trigger-event test and verify the test event is received in the Local WEKA Home portal under the Events section.

  3. If required, go to /var/log/pods and review the relevant log according to the timestamp (for example, wekahome-install-03-08-2023_16-29.log).

hashtag
IPv6 validation

  1. Confirm IPv4 and IPv6 networking: Run the following and confirm both IPv4 and IPv6 addresses are listed in the output.

Example output:

  1. Access LWH using IPv4: Replace <IPv4_address> with the actual IPv4 address of the LWH and run the following command:

Example:

Expected response:

  1. Access LWH using IPv6: Replace <IPv6_address> with the actual IPv6 address of the LWH and run the following command:

Example:

Expected response:

hashtag
Troubleshoot IPv6 issues

  • IPv6 address not assigned:

    • Verify network interface configuration.

    • Check network connectivity with IPv6 ping.

    • Ensure router advertisements are enabled if using SLAAC.

  • Connection failures:

    • Verify firewall rules allow IPv6 traffic.

    • Confirm client machine has global IPv6 connectivity.

    • Check network security group configurations.

hashtag
Upgrade the Local WEKA Home

The upgrade process takes up to 5 minutes. It is recommended to perform the upgrade during a maintenance window.

Certain upgrades require a fresh installation, as direct in-place upgrades are not supported in some cases.

  • Upgrading from minikube or WMS to the Local WEKA Home 3.0 bundle (based on K3s) is not supported. To upgrade, install the new Local WEKA Home bundle on a new server and configure API forwarding from the minikube cluster to the new K3s cluster.

  • IPv6 support requires a fresh installation. It is not possible to upgrade an existing LWH deployment with IPv4 to include IPv6. If IPv6 is needed, install a new LWH instance with dual-stack networking configured during cluster creation.

Procedure

  1. Download the latest Local WEKA Home bundlearrow-up-right to the dedicated physical server (or VM).

  2. Run bash wekahome-*.bundle

  3. To modify the existing configuration, open the /opt/wekahome/config/config.json file and modify the settings. See Deploy Local WEKA Home on K3s.

  4. Run homecli local upgrade. For an upgrade, it takes about 2 minutes.

  5. Run kubectl get pods -n home-weka-io and verify in the results that all pods have the status Running or Completed. To wait for the pods' statuses, run watch kubectl get pods -n home-weka-io.

  6. Verify the Local WEKA Home is upgraded successfully. Run the following command line: helm status wekahome -n home-weka-io

hashtag
Modify the Local WEKA Home configuration

If there is a change in the TLS certificates, SMTP server in your environment, or any other settings in the Local WEKA Home configuration, you can modify the existing config.json with your new settings and apply them.

Procedure

  1. Open the /opt/wekahome/config/config.json file and modify the settings. See Deploy Local WEKA Home on K3s.

  2. Run homecli local upgrade

  3. Run kubectl get pods -n home-weka-io and verify in the results that all pods have the status Running or Completed. To wait for the pods' statuses, run watch kubectl get pods -n home-weka-io.

  4. Verify the Local WEKA Home is updated successfully. Run the following command line: helm status wekahome -n home-weka-io

hashtag
Change the Local WEKA Home listening ports

The LWH uses ports 80 for HTTP and 443 for HTTPS by default. You can reconfigure these ports if they are already in use on the management host. Changing the ports allows LWH to run without conflicting with existing applications.

Procedure

  1. Run the following command to edit the traefik service configuration that manages ingress traffic for LWH:

  2. In the configuration file that opens, locate the entries for port: 80 and port: 443.

  3. Update the port numbers to the alternative ports you require.

  4. Save the file and exit the text editor. For example, in the vi editor, type :wq and press Enter.

The changes take effect immediately.

hashtag
Check Local WEKA Home health

After deploying LWH, it is essential to verify its health to ensure all components are functioning correctly. A healthy LWH means that all pods are running without issues, CPU and memory usage are within acceptable limits, and there are no critical low disk space.

If some pods are restarting frequently, producing errors, or failing to start, LWH is considered unhealthy and may require intervention. The following steps guide you in checking the health status of LWH.

hashtag
Procedure

  1. Check the status of all pods Run the following command to get an overview of the pod statuses:

    • If all pods are running, the system is healthy.

    • If any pods are faulty, proceed with a detailed check.

  2. Get detailed pod status To identify specific faulty pods and their issues, use the following command:

    (For more information on the pod statuses and reasons, see the kubectl documentation.) Example output for a faulty pod:

  3. Check pod status in JSON format (optional) For automated processing, output can be formatted in JSON:

    • A response of { "healthy": true } indicates that all pods are running correctly.

    • If any pod has issues, to get detailed JSON output, use the following command:

      Example output showing a faulty pod:

  4. Verify ingress address response The ingress address in the home-weka-io namespace should return HTTP 200, indicating that the service is reachable:

    • If this check fails, LWH might be experiencing connectivity or service-related issues.

hashtag
Troubleshoot the Local WEKA Home deployment

hashtag
Symptom: browsing to the Local WEKA Home returns an error

The probable cause can be, for example, a communication problem.

hashtag
Resolution

  1. Check the firewall and node IP settings. If you didn't set up a firewall (see Deploy Local WEKA Home on K3s), set valid rules and run:

  1. Retrieve the ingress pod (controller) of the Local WEKA Home.

  1. Retrieve the logs and look for the error.

hashtag
Symptom: when executing any command on the Local WEKA Home, the error “no space left” is displayed

The probable cause for this issue is that the containers dir (/var/lib/rancher/k3s) consumes disk space.

hashtag
Resolution

Do one of the following:

  • Try to clean unused images with homecli local cleanup images

  • Resize the disk and reinstall the Local WEKA Home.

  • Relocate the K3s root directory path to a new path on a larger device (if it exists) and copy the content from the old path to the new path.

hashtag
Symptom: when testing the integration, the email is not received

The probable cause can be issues related to the SMTP server, such as wrong credentials or recipient email address.

hashtag
Resolution

  1. On the Integration page, select Test Integration. Wait until an error appears.

  2. Retrieve the logs and search for the error. On the Local WEKA Home terminal, run the following command:

hashtag
Collect LWH deployment diagnostics

The LWH provides a script that collects various resource details from the LWH deployed on the Kubernetes cluster and generates an archive. This information helps the Customer Success Team and R&D to analyze and provide support when troubleshooting is needed.

The LWH deployment diagnostics provide the following information:

  • Pods status

  • Logs of all pods

  • K3s settings

  • Free disk space

  • CPU and memory usage

  • Name resolutions

  • LWH version

  • Syslogs

Procedure

  1. Run the following command:

  1. Once you generate the LWH deployment diagnostics archive file, send it to the Customer Success Team for analysis.

Parameters

Parameter
Description

archive*

The path and output archive file name. For example: /path/diag/lwh_diagnostics.tar.gz

include-sensitive

Include sensitive data in the archive. For example, value overrides. Use this parameter only if required by the Customer Success Team.

full-disk-scan

Perform a higher level of disk scan. Use this parameter only if required by the Customer Success Team.

verbose

Provide a higher verbosity level of the debug information.

PreviousDeploy Local WEKA Home on K8sNextDeploy Local WEKA Home on Minikube

Last updated