How to manage a cluster

This document describes how to create, configure, and monitor a cluster on the Segura® Platform using Orbit CLI.

The orbit cluster command configures the cluster topology, converges each node to the declared state, and verifies the environment health.

Prerequisites

• Two or more Segura® Platform VMs deployed and running with the same version.
• All VMs with identical or equivalent compute resources (CPU, RAM, disk).
• Full mesh connectivity confirmed between all nodes.
• All appliances individually licensed and operating in standalone mode.
• SSH access to all nodes with sudo privileges to run orbit commands.

Create the cluster

Configure the cluster exclusively through Orbit CLI. Version 4.2.9 discontinued the web interface configuration; do not use this option anymore.

Step 1: Pre-flight check

Before any configuration, run the following checks on all nodes.

1. Confirm all nodes are in standalone mode and running:

   sudo orbit app status
   

   Expected output on each node before cluster configuration:

   Application: Active
   Replication: Inactive
   Instance:    Primary
   Node Name:   <node_hostname>
   Maintenance: No
   Main:        Yes
   

2. Confirm network connectivity between nodes. From each node, verify that the following ports are reachable on the other nodes:

   nmap -p 9300,4567,4248,443 <IP_of_other_node>
   

3. Confirm all nodes are running the same Segura® Platform version:

   sudo orbit version
   

4. Confirm all services are running on all nodes:

   sudo orbit services list
   

   Expected output for all services:

   Active: active (running)
   

Step 2: Configure each node

Run the following command on each node that will be part of the cluster, before applying the configuration on any of them.

sudo orbit cluster configure

The interactive wizard prompts for the following information in this order:

Cluster primary IP:
Secondary node IPs (one per line, blank to finish):
  Secondary IP #1 (blank to finish):
  Secondary IP #2 (blank to finish):
  ...
Arbiter IP (blank for none):

• At Cluster primary IP, enter the primary node IP. Use the same value on all nodes.
• At each Secondary IP prompt, enter the secondary node IPs, one per line, in the same order on all nodes. Leave the next prompt blank to finish the list.
• At Arbiter IP, leave it blank if there is no arbiter in the environment.

When complete, the wizard displays a summary. Confirm the values are correct before proceeding to the next node:

Configuration summary:
  mode       : cluster
  role       : primary
  file sync  : true
  primary    : <node1-ip>
  secondaries: <node2-ip>, <node3-ip>
  arbiter    : (none)

Step 3: Apply the configuration node by node

Only after Step 2 is complete on all nodes, apply the configuration one node at a time, starting with the primary node.

1. On the current node, open a screen session and apply the configuration. This step takes a long time. Wait for it to complete before checking the output:

   sudo screen -S cluster
   sudo orbit cluster apply
   

   Expected output at the end:

   Cluster apply completed on this node. Verify with 'orbit cluster status'.
   

2. Confirm replication completed before moving to the next node:

   grep "Synchronized with group, ready for connections" /var/log/mysql/mysql-error.log
   

   Expected output:

   <current date> [Note] WSREP: Synchronized with group, ready for connections
   

3. Only after confirming both conditions in items 1 and 2 on the current node, proceed to the next node and repeat Step 3.

Check the cluster status

Check the general cluster status:

sudo orbit cluster status

Expected output on the primary node:

Cluster status
  mode           : cluster
  declared role  : primary
  primary node   : <node1-ip>
  this is primary: true
  galera        : healthy (Synced)
  nats          : managed
  elasticsearch : managed
Cluster is healthy.

Check the application status on the primary node:

sudo orbit app status

Expected output on the primary node:

Application: Active
Replication: Active
Instance:    Cluster
Node Name:   cluster1_<id>
Primary:     cluster1_<id>
Main:        Yes

On the secondary node, the output must show:

• Application: Active.
• Replication: Active.
• Instance: Cluster.
• Primary: pointing to the primary node name.
• Main: No.

Check the Elasticsearch status:

sudo orbit elasticsearch status

The output must show "status": "green", with number_of_nodes and number_of_data_nodes matching the total number of nodes, and unassigned_shards: 0.

Troubleshooting

NATS authentication errors during initial cluster formation

During the first 1 to 2 minutes after cluster formation, the nats-senhasegura / nats-qr-senhasegura logs may display messages such as:

[WRN] Auth callout violation
[ERR] ... authentication error - Nkey "..."

This behavior is transient and resolves automatically when the cluster converges. Treat it as a real issue only if auth/nkey errors continue for more than 5 minutes after orbit cluster apply completes. If they do, check the nats-senhasegura service logs on the affected node.

Periodic SSL disconnect messages in the MySQL log

The mysql-error.log file may periodically display (every 30 to 60 seconds):

WSREP: Failed to establish connection: unexpected eof while reading (SSL routines)

This message can appear even on healthy clusters and does not necessarily indicate a replication failure. Before treating it as an issue, confirm the cluster health directly:

sudo mysql -e "SHOW STATUS LIKE 'wsrep_%';"

Verify that:

• wsrep_cluster_status is Primary.
• wsrep_ready is ON.
• wsrep_local_state_comment is Synced.