Expand and Rebalance System | Ocient Documentation

If your system needs more resources, you can expand it by installing additional nodes and drives. The steps for expanding a system depend on the type of node you are adding.

These sections explain the steps for adding each Ocient® node type. Foundation Nodes include additional steps for adding more storage drives and rebalancing the system.

For information on how to replace nodes that have failed or are damaged, see Replace Nodes.

For information on replacing drives that have failed or are damaged, see Replace an NVMe Node Drive.

Prerequisites

These prerequisites apply to adding any node type:

The process requires either the System Administrator role or the granted UPDATE privilege on SYSTEM.
Any new nodes should have the same OS kernel version as other nodes in the system.
Any new nodes should have the same Ocient System version as the other nodes. For details, see Ocient Application Installation.
Ensure that your system meets the requirements outlined in Ocient System Bootstrapping.
Before adding nodes, stop any query or loading process running on your system.

Add Foundation Nodes

A core component of an Ocient System, the Foundation Nodes store user data in Ocient and perform the bulk of query processing. By adding more Foundation Nodes, a system can support extra storage capacity as well as better performance for query processing.

This process requires rebalancing the data to distribute the query processing across all available hardware, including newly installed disks or nodes.

System Considerations

No individual cluster should have more than two petabytes of storage.
Each cluster should have the same number of Foundation Nodes.

Tutorial

Create the /var/opt/ocient/bootstrap.conf file on the new node using this YAML example. This YAML file should follow a similar format and parameters as the bootstrap.conf files on your other nodes.

Replace the <FIRST_NODE_ADDRESS> with the IP or Hostname of a node running the Administrator Role on your system. If you are not using DNS, use nodeAddress instead of adminHost.

Use a user account with System Administrator privileges for your system for the adminUserName and adminPassword.

adminHost: <FIRST_NODE_ADDRESS>
adminUserName: my_admin
adminPassword: example_password

Start the new node by using this command.

sudo systemctl start rolehostd

This process takes about a minute to complete. This step accepts the node into the system, but it has not been assigned a role or a storage cluster.

To validate that the node is on the system, execute this query using the sys.nodes system catalog table.

SELECT name, status
FROM sys.nodes
ORDER BY 1;

Output

name                               status
--------------------------------------------------------------------
admin01                            ACCEPTED
admin02                            ACCEPTED
admin03                            ACCEPTED
loader01                           ACCEPTED
loader02                           ACCEPTED
loader03                           ACCEPTED
foundation01                       ACCEPTED
foundation02                       ACCEPTED
foundation03                       ACCEPTED
foundation04                       ACCEPTED
foundation05                       ACCEPTED
foundation06                       ACCEPTED
foundation07                       ACCEPTED
foundation08                       ACCEPTED
foundation09                       ACCEPTED
foundation10                       ACCEPTED
foundation11                       ACCEPTED
foundation12                       ACCEPTED
foundation_new                     ACCEPTED
sql01                              ACCEPTED
sql02                              ACCEPTED

The output shows the new node, named foundation_new in this example, listed as ACCEPTED in the status column.

Execute this ALTER CLUSTER SQL statement to add the new node to a storage cluster. In this example, replace storage_cluster_1 and foundation_new with your cluster name and node name, respectively.

ALTER CLUSTER "storage_cluster_1"
    ADD PARTICIPANTS "foundation_new";

This example adds only one new node. For information on adding multiple nodes, see ALTER CLUSTER ADD PARTICIPANTS.

Restart the rolehostd process on all nodes in the system by running the following series of commands at the shell terminal.

First, stop the rolehostd process on all nodes.

sudo systemctl kill -s SIGKILL rolehostd

Confirm that the rolehostd process is no longer running.

sudo systemctl status rolehostd

Start the rolehostd process on the system again.

sudo systemctl start rolehostd

You can verify the new node is active by executing this SQL query after you connect to the database.

SELECT n.name,
       ns.operational_status
FROM sys.node_status ns
JOIN sys.nodes n ON ns.node_id = n.id
ORDER BY n.name;
name                               operational_status
------------------------------------------------------------------------------------------
admin01                            Active
admin02                            Active
admin03                            Active
loader01                           Active
loader02                           Active
loader03                           Active
foundation01                       Active
foundation02                       Active
foundation03                       Active
foundation04                       Active
foundation05                       Active
foundation06                       Active
foundation07                       Active
foundation08                       Active
foundation09                       Active
foundation10                       Active
foundation11                       Active
foundation12                       Active
foundation_new                     Active
sql01                              Active
sql02                              Active

Perform a rebalance task to distribute your data across your newly expanded system evenly. For details, see Rebalance System.

Add More Drives to Foundation Nodes

Foundation Nodes can support extra storage by adding additional NVMe drives. This tutorial shows the steps to integrate new drives into an existing Ocient System.

System Considerations

For best performance, all Foundation Nodes should have equal storage capacity.
No individual cluster should have more than two petabytes of storage.

Prerequisites

The process requires systemctl access on your system OS.
Any new NVMe drives added to the system must be blank and unpartitioned.

Tutorial

Shut down the rolehostd process from the shell prompt.

sudo systemctl stop rolehostd

Install the new storage drives in your system.

Restart the node from the shell prompt.

sudo systemctl restart rolehostd

The rolehostd process recognizes the new drive.

To confirm the new drives are active and running, you can connect to your system and query the sys.storage_device_status system catalog table.

To use this example query, replace <NODE_NAME> with the name of the node where you are adding a drive.

SELECT
  n.name AS node_name,
  s.node_id,
  s.id AS serial_number,
  s.pci_address,
  s.device_status,
  s.device_model
FROM
  sys.nodes n
JOIN sys.storage_device_status s
  ON n.id = s.node_id
WHERE
  n.name = '<NODE_NAME>';

Output

|node_name  |node_id                             |serial_number                       |pci_address                                             |device_status|device_model                           |
|-----------|------------------------------------|------------------------------------|--------------------------------------------------------|-------------|---------------------------------------|
|foundation0|9330a0b3-b3b7-4503-949c-043b196c0cc4|6156962f-fcf6-4299-bbd7-2618fc6f1d00|/var/opt/ocient/6156962f-fcf6-4299-bbd7-2618fc6f1d00.dat|ACTIVE       |PCIe Data Center SSD INTEL SSDPE2ME800G4|
|foundation0|9330a0b3-b3b7-4503-949c-043b196c0cc4|e6a4b24f-0d49-4704-b7ce-18af163c0701|/var/opt/ocient/e6a4b24f-0d49-4704-b7ce-18af163c0701.dat|ACTIVE       |PCIe Data Center SSD INTEL SSDPE2ME800G4|

This output lists the serial numbers of all drives in the foundation0 node, including their statuses.

Perform a rebalance task to distribute your data across your newly expanded system evenly. For details, see Rebalance System.

Rebalance System

REBALANCE task execution redistributes your data evenly across your segment groups and clusters. Rebalance your system if you have recently installed new hardware, particularly new nodes or drives.

System Considerations

Currently, the REBALANCE task does not move damaged segment groups. If you suspect that you might have a significant number of damaged segment groups, you can execute these steps to check and fix the groups:

Ensure all nodes are online.
Check whether you have damaged segment groups using the sys.segment_groups system catalog table. If damaged segment groups are present, execute a REBUILD task to rebuild the groups before rebalancing the system. For details, see Guide to Rebuilding Segments.
Execute the REBALANCE task using this tutorial.

You can find information in the sys.degraded_segment_groups system catalog table to identify the damaged segment groups that need fixing. This table shows all segment groups with the DAMAGED or UNAVAILABLE state. You can also check the sys.storage_used system catalog table, which shows approximately the same number of used bytes used_bytes for each node entry with the same table after the rebalance execution.

Prerequisites

Only one rebalance task can execute at a time on the system. The system logs an error if you try to start a second rebalance task.
You must have the System Administrator role, or be granted the UPDATE privilege on SYSTEM.

Tutorial

Execute a SELECT SQL statement to view how the system has distributed the existing data using the sys.tables and sys.nodes system catalog tables.

SELECT t.name AS table_name,
    n.name AS node_name,
    su.used_bytes FROM sys.storage_used su
    JOIN sys.tables t ON t.id = su.table_id
    JOIN sys.nodes n ON n.id = su.node_id
ORDER BY t.name,
    n.name;

Output

+-------------------------+-----------+--------------+
| table_name              | node_name |   used_bytes |
|-------------------------+-----------+--------------|
| table0                  | lts0      |     48000000 |
| table0                  | lts1      |     11000000 |
| table0                  | lts2      |     79000000 |
| table0                  | lts3      |     73000000 |
| table0                  | lts4      |     85000000 |
| table0                  | lts5      |     50000000 |
| table0                  | lts6      |     51000000 |
| table0                  | lts7      |     41000000 |
| table1                  | lts0      |     66000000 |
| table1                  | lts1      |     77000000 |
| table1                  | lts2      |     73000000 |
| table1                  | lts3      |     43000000 |
| table1                  | lts4      |     76000000 |
| table1                  | lts5      |     51000000 |
| table1                  | lts6      |     71000000 |
| table1                  | lts7      |    105000000 |

The query output shows how data is distributed across your nodes.

Execute the REBALANCE task named rebalance_task to reorganize all data in a balanced state using the CREATE TASK SQL statement.

CREATE TASK rebalance_task TYPE REBALANCE;

For details about the syntax for creating tasks, see Distributed Tasks.

View the status of the tasks as the rebalance operation executes using the sys.subtasks system catalog table.

SELECT id,
    name,
    start_time,
    task_type,
    execution_type,
    status,
    details
FROM sys.subtasks
WHERE task_type = 'rebalance'
ORDER BY start_time DESC;

Output

+--------------------------------------+--------+----------------------------+-------------+-------------------------+----------+--------------------------------------------------+
| id                                   | name   | start_time                 | task_type   | execution_type          | status   | details                                          |
|--------------------------------------+--------+----------------------------+-------------+-------------------------+----------+--------------------------------------------------|
| ed68a070-d7f7-4c50-9b94-84b04b4503c2 |        | 2024-04-17 21:44:20.268000 | rebalance   | intra_cluster_rebalance | COMPLETE |                                                  |
| 9e6c3374-6e9c-4a84-9bf8-ba1db3d866eb |        | 2024-04-17 21:44:20.268000 | rebalance   | intra_cluster_rebalance | COMPLETE |                                                  |
| aebb75d8-e066-4813-b7e8-c272e5a3b8e9 |        | 2024-04-17 21:44:14.768000 | rebalance   | inter_cluster_rebalance | COMPLETE |                                                  |
| 5cec5095-8b3a-4868-a10d-35fe6a32e998 |        | 2024-04-17 21:44:14.564000 | rebalance   | NULL                    | COMPLETE | Task finalizing due to terminal status: COMPLETE |
+--------------------------------------+--------+----------------------------+-------------+-------------------------+----------+--------------------------------------------------+

The output shows when the REBALANCE task is finished.

When the REBALANCE task runs, segments are in the REBUILDING state. After the Ocient System completes this task, all segments should transition to the INTACT state.

Execute the query from Step 1 again to see how the system reorganized the data across nodes.

SELECT t.name,
    n.name,
    su.used_bytes
FROM sys.storage_used su
    JOIN sys.tables t ON t.id = su.table_id
    JOIN sys.nodes n ON n.id = su.node_id
ORDER BY t.name,
    n.name;

Output

+-------------------------+----------+--------------+
| name                    | name_1   |   used_bytes |
|-------------------------+----------+--------------|
| table0                  | lts0     |     48000000 |
| table0                  | lts1     |     39000000 |
| table0                  | lts2     |     71000000 |
| table0                  | lts3     |     53000000 |
| table0                  | lts4     |     85000000 |
| table0                  | lts5     |     50000000 |
| table0                  | lts6     |     51000000 |
| table0                  | lts7     |     41000000 |
| table1                  | lts0     |     66000000 |
| table1                  | lts1     |     77000000 |
| table1                  | lts2     |     73000000 |
| table1                  | lts3     |     43000000 |
| table1                  | lts4     |     76000000 |
| table1                  | lts5     |     68000000 |
| table1                  | lts6     |     71000000 |
| table1                  | lts7     |     88000000 |

The output shows the redistributed data.

The REBALANCE task redistributes data across segment groups, so there can be some variance in data across nodes.

Add Loader Nodes

Adding more Loader Nodes to your system can improve the throughput of data loading and resiliency against loading failure. By having extra Loader Nodes, you can also dedicate sets of nodes for specific pipelines.

System Considerations

Before starting this process, ensure you meet the requirements in the Prerequisites section.

Tutorial

Stop any active data pipelines. Execute this SQL statement by replacing pipeline_name with the name of your data pipeline.

STOP PIPELINE <pipeline_name>;

Create the /var/opt/ocient/bootstrap.conf file on the new Loader Node using this YAML example. This YAML file should follow a similar format and parameters to the bootstrap.conf file on your other nodes.

Replace the <FIRST_NODE_ADDRESS> with the IP address or hostname of a node running the Administrator Role on your system. If you are not using a DNS, use nodeAddress instead of adminHost.

Specify a user account with System Administrator privileges for your system for the username adminUserName and password adminPassword.

adminHost: <FIRST_NODE_ADDRESS>
adminUserName: my_admin
adminPassword: example_password

Start the new node by using this command.

sudo systemctl start rolehostd

This process takes about a minute to complete. This step accepts the node into the system, but does not assign a role.

Add the streamloader role to the new node.

Execute this SQL statement by replacing <new_node_name> with the new node name.

ALTER NODE <new_node_name> ADD ROLE streamloader;

Restart the rolehostd process on the replacement node by running this command at the shell terminal on the replacement node.

sudo systemctl restart rolehostd

Restart the pipeline. Execute this SQL statement by replacing pipeline_name with the name of your data pipeline. Optionally, specify the Loader Node by name with the USING LOADERS keywords to prioritize it for this pipeline.

START PIPELINE <pipeline_name> USING LOADERS <new_node_name>;

If you use the legacy LAT service, you must stop loading and copy the LAT configuration files to any new Loader Nodes. For details, see Configure the LAT Service.

This step is unnecessary for systems that use Ocient data pipelines for loading.

Add SQL Nodes

Adding more SQL Nodes to your system can improve query optimization and processing, particularly for aggregation and join operations. Extra SQL Nodes also provide system resiliency, especially when assigned the admin role.

System Considerations

Before starting this process, ensure you meet the requirements in the Prerequisites section.

Tutorial

Create the /var/opt/ocient/bootstrap.conf file on the new SQL Node using this YAML example. This YAML file should follow a similar format and parameters to the bootstrap.conf file on your other nodes.

Replace the <FIRST_NODE_ADDRESS> with the IP address or hostname of a node running the Administrator Role on your system. If you are not using a DNS, use nodeAddress instead of adminHost.

Specify a user account with System Administrator privileges for your system for the username adminUserName and password adminPassword.

adminHost: <FIRST_NODE_ADDRESS>
adminUserName: my_admin
adminPassword: example_password

Start the new node by using this command.

sudo systemctl start rolehostd

This process takes about a minute to complete. This step accepts the node into the system, but does not assign a role.

Add the sql role to the new node.

Execute this SQL statement by replacing <new_node_name>with the new node name.

ALTER NODE <new_node_name> ADD ROLE sql;

Optionally, you can also assign the admin role to the SQL Node. At least one SQL Node must always fulfill this role. By default, the system assigns the admin role to the first SQL Node in the system.

For details about the admin role, see Node Configuration with the Administrator Role.

Execute this SQL statement by replacing <new_node_name> with the name of your new node.

ALTER NODE <new_node_name> ADD ROLE admin;

Restart the rolehostd process on the replacement node by running this command at the shell terminal on the replacement node.

sudo systemctl restart rolehostd

Assign the new SQL Node to a connectivity pool with the ALTER CONNECTIVITY_POOL statement.

In this example, the statement assigns the SQL Node sql2 to the connectivity pool cp1 with the IP address 111.1.1.1 and port number 4050 for listening. Specify the local IP address and port number 4050 to return to the client.

ALTER CONNECTIVITY_POOL cp1
    ADD PARTICIPANTS(
        NODE sql2
        LISTEN_ADDRESS '111.1.1.1'
        LISTEN_PORT 4050
        ADVERTISED_ADDRESS 'localhost'
        ADVERTISED_PORT 4050);

Replace Nodes

Replace an NVMe Node Drive

System Catalog

Prerequisites

Add Foundation Nodes

System Considerations

Tutorial

Add More Drives to Foundation Nodes

System Considerations

Prerequisites

Tutorial

Rebalance System

System Considerations

Prerequisites

Tutorial

Add Loader Nodes

System Considerations

Tutorial

Add SQL Nodes

System Considerations

Tutorial

Related Links