Guide to Rebuilding Segments

Data in an Ocient® System is erasure coded to provide resilience to disk and Foundation Node failures. The resilience of a system depends on the parity width of the storage space, which represents the number of associated disks (or nodes) that can fail without interruption of service or data loss.

For details about storage spaces and parity width, see Configure Storage Spaces.

Data Segment Statuses

You can check the status of your system segment groups by querying the sys.segment_groups system catalog table. For details, see System Catalog.

This table describes the states for data segments.

Recovery Considerations

If a segment has the DAMAGED or MISSING status, queries can proceed by reconstructing the data on demand using the remaining erasure-coded data in the segment group. Having a non-INTACT status means that input or output (I/O) performance is significantly reduced.

To restore full performance, you need to run a segment rebuild task. Segment rebuilding is not automatic. The system administrator must manually invoke it.

A segment rebuild fails, and data is lost completely if the number of segments with the DAMAGED status in a segment group exceeds the parity width of the storage space. To avoid data loss:

• Provision the parity width of the storage space at or above the number of expected concurrent node failures.
• Rebuild damaged or permanently missing segments as soon as possible.

Checking for Abnormal Segments

You can find any segments that need a rebuild by querying for segment groups with an abnormal status.

Examples

Finding Faulty Segment Groups

This example query finds any segment groups with the DAMAGED or MISSING status.

SELECT * FROM sys.segment_groups WHERE status IN ('DAMAGED', 'MISSING');

Output

|         "id"         |              "cluster_id"              | "segment_type" | "status" | "primary_owner" |              "loader_id"               |               "table_id"               |               "scope_id"               | "block_size" | "begin_time" | "end_time" | "coding_algorithm" | "coding_block_size" | "coding_threshold" | "coding_width" | "replication" | "parity_cycle" |     "created_time"      | "rolehostd_version" |               "commit_hash"                |    "timestamp"    | "build_user" | "depth" | "removal_type" |
|----------------------|----------------------------------------|----------------|----------|-----------------|----------------------------------------|----------------------------------------|----------------------------------------|--------------|--------------|------------|--------------------|---------------------|--------------------|----------------|---------------|----------------|-------------------------|---------------------|--------------------------------------------|-------------------|--------------|---------|----------------|
| "720576045199095878" | "074c32ec-4f92-4718-ada1-5eb55eafdcb3" | TKT_SEGMENT    | DAMAGED  |                 | "9a8f7ea7-613c-499f-ab2d-937ab0ce992e" | bd18d33b-aae9-4be2-a2f4-76ecdc34c2ba   | "8bc119d2-875a-47e4-b016-87fde83e77d6" |         4096 |            0 |          1 | XOR_PARITY         |                4096 |                  2 |              3 |             1 |              1 | 2025-02-26 20:23:34.274 | "25.0.0"            | "91ab76ae57491ade0122bc7b594d5c3c6e0bf40c" | "20250109.221519" |              |       0 | NOT_REMOVED    |
| "716072445571725367" | "074c32ec-4f92-4718-ada1-5eb55eafdcb3" | TKT_SEGMENT    | DAMAGED  |                 | "9a8f7ea7-613c-499f-ab2d-937ab0ce992e" | "9fe79009-ee75-4429-b29b-3a614a166751" | a8c805d0-2144-46a7-8374-52cb88d64244   |         4096 |            0 |          1 | XOR_PARITY         |                4096 |                  2 |              3 |             1 |              1 | 2025-02-26 20:23:19.447 | "25.0.0"            | "91ab76ae57491ade0122bc7b594d5c3c6e0bf40c" | "20250109.221519" |              |       0 | NOT_REMOVED    |

Finding Clusters and Nodes with Faulty Segment Groups

Inspect the count of damaged groups by cluster and node.

SELECT
  c.name AS cluster_name,
  n.name AS node_name,
  g.status AS segment_group_status,
  seg.status AS segment_status,
  seg.kind,
  COUNT(*) AS segment_count
FROM
  sys.segment_groups g
LEFT JOIN
  sys.clusters c
  ON c.id = g.cluster_id
LEFT JOIN
  sys.stored_segments seg
  ON seg.segment_group_id = g.id
LEFT JOIN
  sys.nodes n
  ON n.id = seg.node_id
WHERE
  g.status <> 'INTACT'
  AND (seg.status <> 'INTACT' OR seg.status IS NULL)
  GROUP BY 1,2,3,4,5
  ORDER BY 1,2,3,4,5;

Output

|   "cluster_name"   | "node_name" | "segment_group_status" | "segment_status" | "kind"  | "segment_count" |
|--------------------|-------------|------------------------|------------------|---------|-----------------|
| foundation_cluster | foundation0 | DAMAGED                |                  | VIRTUAL |               2 |

Starting a Segment Rebuild Task

A user with System Administrator privileges can start a segment rebuild using the CREATE TASK TYPE REBUILD SQL statement. You cannot cancel a segment rebuild task after it is started.

Most commonly, a rebuild task repairs all damaged or missing segments across the system.

Example

Create a rebuild task.

CREATE TASK TYPE REBUILD;

The system can continue to perform queries while rebuilding segments, but the process can impact I/O performance.

Advanced Rebuild Commands

Rebuild tasks can also execute on specific Foundation Nodes or clusters. For information on fine-tuning rebuild tasks, see CREATE TASK.

Checking Rebuild Task Status

Monitor the status of current and past segment rebuild tasks from the sys.subtasks system catalog table. For details, see System Catalog.

SELECT * FROM sys.subtasks WHERE task_type = 'rebuild';

This table describes the statuses for a rebuild task.

Related Links

CREATE TASK

Erasure Coding