Ctrlk
For the complete documentation index, see llms.txt. This page is also available as Markdown.

Hardware maintenance

Hardware maintenance operations ensure cluster reliability and performance through systematic component management and failure response procedures. These operations span from routine preventive maintenance to critical component replacements.

Key operations:

  • Node management

    • Graceful and forced node reboots.

    • Node replacement and removal.

    • Complete rack decommissioning procedures.

  • Container operations

    • Container migration from failed nodes.

    • Container replacement on active nodes.

    • Container management on denylisted nodes.

  • Storage management

    • Drive replacement in converged setups.

    • Storage integrity verification.

    • Component failure recovery.

Each procedure follows established protocols to maintain system stability and minimize service disruption during maintenance activities. The documented procedures enable administrators to execute both planned maintenance and emergency responses while preserving data integrity and system performance.

Before you begin

Before performing any hardware maintenance or replacement tasks, ensure you have:

  • Administrative access to your Kubernetes cluster.

  • SSH access to the cluster nodes.

  • kubectl command-line tool installed and configured.

  • Proper backup of any critical data on the affected components.

  • Required replacement hardware (if applicable).

  • Maintenance window scheduled (if required).

Perform standard verification steps

This procedure describes the standard verification steps for checking WEKA cluster health. Multiple procedures in this documentation refer to these verification steps to confirm successful completion of their respective tasks.

Procedure

  1. Log in to the wekacontainer:

  1. Check the WEKA cluster status:

Example

  1. Check cluster containers.

Example

  1. Check the WEKA filesystem status.

Example

  1. Verify the status of the WEKA cluster processes is UP.

Example

  1. Check all pods are up and running.

Example

Force reboot a machine

A force reboot may be necessary when a machine becomes unresponsive or encounters a critical error that cannot be resolved through standard troubleshooting. This task ensures the machine restarts and resumes normal operation.

This procedure assumes a dedicated topology where backend nodes (running drive and compute pods) and client nodes run on separate Kubernetes nodes. If your cluster uses an Axon topology where backend and client pods run on the same nodes, do not perform phases 6–8.

For a worker node in Axon topology, complete phases 1–5 for each node you want to reboot. Draining the node evicts all WEKA pods in a single pass.

Procedure

Phase 1: Perform standard verification steps.

Phase 2: Cordon and evict backend k8s nodes.

To cordon and evict a node, run the following commands. Replace <k8s_node_IP> with the target k8s node's IP address.

  1. Cordon the backend k8s node:

Example:

  1. Evict the running pods ensuring data is removed. For example, drain the backend k8s node:

Example

  1. Validate node status:

Example

  1. Verify pod statuses across namespaces:

Example

Phase 3: Ensure the WEKA containers are marked as drained.

  1. List the cluster backend containers. Run the following command to display the current status of all WEKA containers in the k8s nodes:

  2. Check the status of the WEKA containers. In the command output, locate the STATUS column for the relevant containers. Verify that it displays DRAINED for the host and backend container.

Example

Phase 4: Force a reboot on all backend k8s nodes. Use the reboot -f command to force a reboot on each backend k8s node.

Example:

After running this command, the container restarts immediately. Repeat for all k8's nodes one by one in your environment.

Phase 5: Uncordon the backend k8s node and verify WEKA cluster status.

  1. Uncordon the backend k8s node:

Example:

  1. Access the WEKA Operator in the backend k8s node:

  1. Verify the weka drives status:

Example

  1. Hardware maintenance.

Ensure all the pods, weka containers and the cluster is in a healthy state (Fully Protected) and IO operations are running (STARTED). Monitor the redistribution progress and alerts.

The following phases (6-8) do not apply to Axon topology.

Phase 6: Cordon and drain all client k8s nodes.

To cordon and drain a node, run the following commands. Replace <k8s_node_IP> with the target k8s node's IP address.

  1. Cordon the client k8s node to mark it as unschedulable:

Example:

  1. Evict the the workload. For example: Drain the client k8s node to evict running pods, ensuring data is removed:

Example

  1. Force reboot all client nodes. Example for one client k8s node:

  1. After the client k8s nodes are up, uncordon the client k8s node. Example for one client k8s node:

Example:

  1. Verify that after uncordoning all client Kubernetes nodes:

  • All regular pods remain scheduled and running on those nodes.

  • All client containers within the cluster are joined and operational.

  • Only pods designated for data I/O operations are evicted.

Example

Phase 7: Force a reboot on all client k8s nodes. Use the reboot -f command to force a reboot on each client k8s node.

Restart only one backend container at a time. Ensure it reaches Ready status before moving on to the next.

Example for one client k8s node:

After running this command, the client node restarts immediately. Repeat for all client nodes in your environment.

Phase 8: Uncordon all client k8s nodes.

  1. Once the client k8s nodes are back online, uncordon them to restore their availability for scheduling workloads. Example command for uncordoning a single client k8s node:

Example

  1. Verify pod status across all k8s nodes to confirm that all pods are running as expected:

  1. Validate WEKA cluster status to ensure all containers are operational:

See examples in Perform standard verification steps.

Remove a rack or Kubernetes node

Removing a rack or Kubernetes (k8s) node is necessary when you need to decommission hardware, replace failed components, or reconfigure your cluster. This procedure guides you through safely removing nodes without disrupting your system operations.

Before you begin

  1. Verify a spare backend node exists and is labeled:

  1. Confirm at least one node beyond the minimum cluster size appears in the output before proceeding. If no spare exists, the removal stalls silently and the operator drains the target node but have available targets to reschedule the containers.

  2. Verify spare signed NVMe drives are available on the spare node before initiating removal, as the operator attempts to provision drives on the replacement node immediately after rescheduling.

Procedure

  1. Create failure domain labels for your nodes:

    1. Label nodes with two machines per failure domain:

      b. Label nodes with one machine per failure domain:

  2. Apply the NoSchedule taint to nodes in failure domains:

  3. Remove WEKA labels from the untainted node:

  4. Configure the WekaCluster:

    1. Export the existing WekaCluster manifest to a file named cluster.yaml:

    2. Update the spec section in cluster.yaml with the failure domain label:

    3. Apply the configuration:

  5. Verify failure domain configuration:

    1. Check container distribution across failure domains using the WEKA cluster container.

    2. Test failure domain behavior by draining nodes which have same FD :

    3. Reboot the drained nodes:

    4. Monitor workload redistribution: Check that workloads are redistributed to other failure domains while nodes in one FD are down

Expected results

After completing this procedure:

  • Your nodes are properly configured with failure domains.

  • Workloads are distributed according to the failure domain configuration.

  • The system is ready for node removal with minimal disruption.

Troubleshooting

If workloads do not redistribute as expected after node drain:

  1. Check node labels and taints.

  2. Verify the WekaCluster configuration.

  3. Review the Kubernetes scheduler logs for any errors.

Perform a graceful node reboot on client nodes

A graceful node reboot ensures minimal service disruption when you need to restart a node for maintenance, updates, or configuration changes. The procedure involves cordoning the node, draining workloads, performing the reboot, and then returning the node to service.

Procedure

  1. Cordon the Kubernetes node to prevent new workloads from being scheduled:

  1. Drain the node to safely evict all pods:

The system displays warnings about DaemonSet-managed pods being ignored. This is expected behavior.

  1. Verify the node status shows as SchedulingDisabled:

  1. Reboot the target node:

  1. Wait for the node to complete its reboot cycle and return to a Ready state:

  1. Uncordon the node to allow new workloads to be scheduled:

  1. Verify that pods are running correctly on the node:

See examples in Perform standard verification steps.

Expected results

After completing this procedure:

  • The node has completed a clean reboot cycle.

  • All pods is rescheduled and running.

  • The node is available for new workload scheduling.

Troubleshooting

If pods fail to start after the reboot:

  1. Check pod status and events using kubectl describe pod <pod-name>.

  2. Review node conditions using kubectl describe node <node-ip>.

  3. Examine system logs for any errors or warnings.

Replace a drive in a converged setup

Drive replacement is necessary when hardware failures occur or system upgrades are required. Following this procedure ensures minimal system disruption while maintaining data integrity.

Before you begin

  • Ensure a replacement drive ready for installation.

  • Identify the node and drive that needs replacement.

  • Ensure you have the necessary permissions to execute Kubernetes commands

  • Back up any critical data if necessary.

Procedure

  1. List and record drive information:

    1. List the available drives on the target node:

    2. Identify the serial ID of the drives:

    3. Record the current drive configuration:

    4. Save the serial ID of the drive being replaced for later use.

Example

  1. Remove node label: Remove the WEKA backend support label from the target node:

Example

  1. Delete drive container: Delete the WEKA container object associated with the drive. Then, verify that the container pod enters a pending state and the drive is removed from the cluster.

Example

  1. Sign the new drive:

    1. Create a YAML configuration file for drive signing:

    2. Apply the configuration:

Example

  1. Block the old drive:

    1. Create a YAML configuration file for blocking the old drive:

    2. Apply the configuration:

Example

  1. Restore node label: Re-add the WEKA backend support label to the node:

Example

  1. Verify the replacement:

    1. Check the cluster drive status:

    2. Verify that:

      • The new drive appears in the cluster.

      • The drive status is ACTIVE.

      • The serial ID matches the replacement drive.

Troubleshooting

  • If the container pod remains in a pending state, check the pod events and logs.

  • If drive signing fails, verify the device path and node selector.

  • If the old drive remains visible, ensure the block operation completed successfully.

  • Maintain system stability by replacing one drive at a time.

  • Keep track of all serial IDs involved in the replacement process.

  • Monitor system health throughout the procedure.

Replace a Kubernetes node

This procedure enables systematic node replacement while maintaining cluster functionality and minimizing service interruption, addressing performance issues, hardware failures, or routine maintenance needs.

Prerequisites

  • Identification of the node to be replaced.

  • A new node prepared for integration into the cluster.

Procedure

  1. Remove node deployment label: Remove the existing label used to deploy the cluster from the node:

Example

  1. List existing WEKA containers to identify containers on the node:

Example

  1. Delete the compute and drive containers specific to the node:

Example

  1. Verify container deletion:

    1. Verify containers are in PodNotRunning status.

    2. Confirm no containers are running on the old node.

      Look for:

      • STATUS column showing PodNotRunning.

      • No containers associated with the old node.

Example

  1. Add backend label to new node: Label the new node to support backends:

Example

  1. Sign drives on new node:

    1. Create a WekaManualOperation configuration to sign drives:

    2. Apply the configuration:

Example

  1. Verification steps:

    1. Verify WEKA containers are rescheduled.

    2. Check that new containers are running on the new node's IP.

    3. Validate cluster status using WEKA CLI.

    For details, see Hardware maintenance.

Non-functional node replacement: When a node becomes unresponsive or faulty, delete the non-functional node: kubectl delete node <node-name>

Kubernetes automatically handles the following:

  • Detects node failure.

  • Removes affected containers.

  • Reschedules containers to available nodes.

Troubleshooting

If containers fail to reschedule, check:

  • Node labels

  • Drive signing process

  • Cluster resource availability

  • Network connectivity

Remove WEKA container from a failed node

Removing a WEKA container from a failed node is necessary to maintain cluster health and prevent any negative impact on system performance. This procedure ensures that the container is removed safely and the cluster remains operational.

Procedure: Remove WEKA container from an active node

To remove a WEKA container when the node is responsive, run the following:

Procedure: Remove WEKA container from a failed node (unresponsive)

  1. Apply the configuration:

  2. If the resign drives operation fails with the error "container node is not ready, cannot perform resign drives operation", set the skip flag:

  3. Wait for the pod to enter the Terminating state.

If the failed node is removed from the Kubernetes cluster, the WEKA container and corresponding stuck pod are automatically removed.

Resign drives manually

If you need to manually resign specific drives, create and apply the following YAML configuration:

Example: wekacontainer conditions added on deletion

Verification

You can verify the removal process by checking the WEKA container conditions. A successful removal shows the following conditions in order:

  1. ContainerDrivesDeactivated

  2. ContainerDeactivated

  3. ContainerDrivesRemoved

  4. ContainerRemoved

  5. ContainerDrivesResigned

Replace a container on an active node

Replacing a container on an active node allows for system upgrades or failure recovery without shutting down services. This procedure ensures that the replacement is performed smoothly, keeping the cluster operational while the container is swapped out.

Procedure

Phase 1: Delete the existing container

  1. Identify the container to be replaced:

  2. Delete the selected container:

Never force-delete WEKA pods. Force deletion removes the pod from the orchestration layer only, while the underlying container continues running untracked. See Deletion behavior.

Example

Phase 2: Monitor deactivation process

  1. Verify that the container and its drives are being deactivated:

    Expected status: The container shows DRAINED (DOWN) under the STATUS column.

  2. Check the process status:

    Expected status: The processes associated with the container show DOWN status.

  3. For drive containers, verify drive status:

    Look for:

    • Drive status changes from ACTIVE to FAILED for the affected container.

    • All other drives remain ACTIVE.

Example

Phase 3: Monitor container recreation

  1. Watch for the new container creation:

  2. Verify the new container's integration with the cluster:

    Expected result: A new container appears with UP status.

  3. Verify the new container's running status:

    Expected status: Running.

  4. Confirm the container's integration with the WEKA cluster:

    Expected status: UP.

  5. For drive containers, verify drive activity:

    Expected status: All drives display ACTIVE status.

See examples in Perform standard verification steps.

Troubleshooting

If the container remains in erminating state:

  1. Check the container events:

  2. Review the operator logs for error messages.

  3. Verify resource availability for the new container.

For failed container starts, check:

  • Node resource availability

  • Network connectivity

  • Service status

Replace a container on a denylisted node

Replacing a container on a denylisted node is necessary when the node is flagged as problematic and impacts cluster performance. This procedure ensures safe container replacement, restoring system stability.

Never force-delete WEKA pods. Force deletion removes the pod from the orchestration layer only, while the underlying container continues running untracked.

Procedure

  1. Remove the backend label from the node that is hosting the WEKA container (for example, weka.io/supports-backends) to prevent it from being chosen for the new container

Example

  1. Delete the pod containing the WEKA container. This action prompts the WEKA cluster to recreate the container, ensuring it is not placed on the labeled node.

Never force-delete WEKA pods. Force deletion removes the pod from the orchestration layer only, while the underlying container continues running untracked. See Deletion behavior.

  1. Monitor the container recreation and pod scheduling status. The container remains in a pending state due to the label being removed.

Example

Expected results

  • The container pod enters Pending state.

  • Pod scheduling fails with message: "nodes are available: x node(s) didn't match Pod's node affinity/selector".

  • The container is prevented from running on the denied node.

Troubleshooting

If the pod schedules successfully on the denied node:

  • Verify the backend support label was removed successfully.

  • Check node taints and tolerations.

  • Review pod scheduling policies and constraints.

PreviousObservability and monitoringNextCluster scaling

Last updated