Set up the Weka-mon external monitoring
The Weka-mon is an external monitoring package, which provides enriched monitoring capabilities using the Grafana and Prometheus tools.
The package also includes the following components:
- Exporter: The Exporter gets the data from the Weka cluster and sends the data to Prometheus.
- Quota Export: The Quota Export manages the quotas and sends the data to Prometheus.
- Alert Manager: The Alert Manger alerts users through an SMTP server when they reach their soft quota limits.
You can set up the Weka-mon package regardless of the data monitoring provided by the Weka GUI.
One of the advantages of setting up the Weka-mon package is that if you already use the Grafana and Prometheus tools for monitoring other products, you can integrate these tools with Weka to correlate and display monitoring information from all your products on the same dashboard.
Weka enriched monitoring data on the Grafana dashboard example
Setting up a dedicated management server (or VM) for the installation is recommended.
- 4 cores
- 16 GB RAM
- 50 GB / partition (for the root)
- 50 GB /opt/ partition (for weka-mon installation)
- 1 Gbps network
- Docker is the recommended container for the Weka-mon setup. To use Docker, the following must be installed on the dedicated management server:
docker-compose-plugindepending on the existing operating system.
To enable communication between the management server and the Weka cluster, the security token is required in the auth-token.json file.
Create the directory
~/.wekain the management server.
The Weka-mon package resides on the GitHub repository. Obtain the Weka-mon package using one of the following methods:
- 2.On the latest release section, select the Source Code link to download.
- 3.Copy the downloaded source code to the dedicated management server or VM and unpack it.
Run the following commands to clone the weka-mon package from GitHub:
git clone https://github.com/weka/weka-mon
install.shscript creates a few directories and sets their permissions.
Run the following command:
The Weka-mon and the exporter configuration are defined in the
- 1.Change the directory to
weka-monand open the
- 2.In the cluster section under the hosts list, replace the hostnames with the actual hostnames/IP addresses of the Weka containers (up to three would be sufficient). Ensure the hostnames are mapped to the IP addresses in /etc/hosts.
4. Optional. Add custom panels to Grafana containing other metrics.
All other settings in the
export.ymlfile have pre-defined defaults that do not need modification to work with weka-mon. All the configurable items are defined but marked as comments by an asterisk (#).
To add custom panels to Grafana containing other metrics from the cluster, you can remove the asterisk from the required metrics (uncomment).
Example: In the following snippet of the
export.yml, to enable getting the FILEATOMICOPEN_OPS statistic, remove the
#character at the beginning of the line.
If the statistic you want to get is in a Category that is commented out, also uncomment the Category line (the first line in the example). Conversely, insert the # character at the beginning of the line to stop getting a statistic.
'ops_driver': # Category
# 'FILEATOMICOPEN_LATENCY': 'microsecs'
# 'FILEATOMICOPEN_OPS': 'ops'
The Weka-mon deployment includes a dedicated container named quota-export. The container includes an Alert Manager that emails users when they reach their soft quota.
The configuration of the quota-export container is defined in the
- 1.Run the following command:
docker-compose up -d
2. Verify that the containers are running using the following command:
[[email protected] weka-mon] 2022-12-05 17:30:37 $ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
ec1d2584acab grafana/loki:2.3.0 "/usr/bin/loki -conf…" 20 minutes ago Up 20 minutes 0.0.0.0:3100->3100/tcp, :::3100->3100/tcp weka-mon_loki_1
4645533501f0 grafana/grafana:latest "/run.sh" 20 minutes ago Up 20 minutes 0.0.0.0:3000->3000/tcp, :::3000->3000/tcp weka-mon_grafana_1
d930e903b74e wekasolutions/export:latest "/weka/export -v" 20 minutes ago Up 7 minutes 0.0.0.0:8001->8001/tcp, :::8001->8001/tcp weka-mon_export_1
dc5f9f710997 wekasolutions/quota-export:latest "/weka/quota-export" 20 minutes ago Up 7 minutes 0.0.0.0:8101->8101/tcp, :::8101->8101/tcp weka-mon_quota-export_1
17689ac9377d prom/prometheus:latest "/bin/prometheus --s…" 20 minutes ago Up 20 minutes 0.0.0.0:9090->9090/tcp, :::9090->9090/tcp weka-mon_prometheus_1
[[email protected] weka-mon] 2022-12-05 17:35:46 $
If the status of the containers is not up, check the logs and troubleshoot accordingly. To check the logs, run the following command:
docker logs <container id>
Once all containers are running, you can connect to Grafana on port 3000 of the management server running the docker containers. The default credentials for Grafana are
If you already have Grafana and Prometheus running in your environment, you only need to run the exporter and add it to the Prometheus configuration.
Do one of the following:
Get and run the container (the
export.ymlconfiguration file is already edited).
The following example maps the
export.ymlconfiguration file in several volumes in the container:
~/.weka directoryto enable the container to read the authentication file.
/dev/logto enable entries in the Syslog.
/etc/hoststo enable the hostname resolution (a DNS can also be used, if exists in the docker environment).
There are more options; you can run the command with
-hfor a full description.
# get the container from dockerhub:
docker pull wekasolutions/export
# example of how to run the container
docker run -d --network=host \
--mount type=bind,source=/root/.weka/,target=/weka/.weka/ \
--mount type=bind,source=/dev/log,target=/dev/log \
--mount type=bind,source=/etc/hosts,target=/etc/hosts \
--mount type=bind,source=$PWD/export.yml,target=/weka/export.yml \
tar xvf export-1.3.0.tar
- 1.Do one of the following:
- 2.Install the required python modules by running the following command:
pip3 install -r requirements.txt
The exporter section defines the program behavior.
# exporter section
The exporter always tries to allocate one host per thread but does not exceed the maximum processes specified in the
max_procsparameter. In a cluster with 1000 hosts, it doubles or triples up the hosts on the threads.