LAT Metrics | Ocient Documentation

Data Pipelines are now the preferred method for loading data into the Ocient System. For details, see Load Data.

Configuration

LAT metrics are exposed using the LAT REST API.

Endpoints

LAT exposes two metrics endpoints; a partitions endpoint, and a pipeline endpoint.

Partition Metrics

The partitions endpoint exposes individual metrics for every partition that is participating in the current LAT instance. For Apache® Kafka® loads, partitions correspond to Kafka’s partitions. For File Source based loads, partitions correspond to the independent file sets created within each File Group for parallel processing.

The partitions metrics endpoint can be accessed by issuing a curl request as follows:

curl http://127.0.0.1:8080/v2/metrics/read/lat:type=partitions

A response for this request will look as follows, with one entry per partition in the partitions array:

{
    "request": {
        "mbean": "lat:type=partitions",
        "type": "read"
    },
    "value": {
        "partitions": [
            {
                "offsets_durable": 0,
                "pushes_errors": 0,
                "pushes_attempts": 0,
                "rows_pushed": 0,
                "offsets_written": 0,
                "records_buffered": 0,
                "records_errors_column": 0,
                "records_errors_deserialization": 0,
                "records_errors_transformation": 0,
                "offsets_processed": 0,
                "lag": 0,
                "partition": "topic-0",
                "records_filter_accepted": 0,
                "records_errors_row": 0,
                "records_filter_rejected": 0,
                "records_errors_generic": 0,
                "producer_send_attempts": 0,
                "offsets_pushed": 0,
                "pushes_unacknowledged": 0,
                "invalid_state": 0,
                "bytes_pushed": 0,
                "errors_partition": 0,
                "records_errors_total": 0,
                "offsets_buffered": 0,
                "complete": 0,
                "offsets_end": 0,
                "producer_send_errors": 0
            }
        ]
    },
    "timestamp": 1642497992,
    "status": 200
}

Partition Metrics Definitions

Metrics definitions are as follows:

Metric	Description
`partition`	Topic partition pair (e.g., `mytopic-0`)
`records_errors_deserialization`	Total number of processed records which failed to deserialize
`records_errors_transformation`	Total number of processed records which failed to transform
`records_errors_column`	Total number of processed records which failed to bind transformed values to columns
`records_errors_row`	Total number of processed records which failed during builder row advancement/other builder errors
`records_errors_generic`	Total number of processed records which failed for any other reason
`records_errors_total`	Sum of all record errors for this topic
`records_buffered`	Total number of processed records which have not yet been pushed. Should always be modulo configured buffer size
`records_filter_rejected`	Number of records rejected by the topic filter.
`records_filter_accepted`	Number of records accepted by the topic filter.
`offsets_processed`	For Kafka loading, records are processed if their offsets are strictly less than the value of `offsets_processed`. For file loading, this represents the index of the most recently processed file. If there is an error during the processing of a record/file, this metric is still updated according to that record/file’s offset. When a file load is complete, this offset becomes equal to offsets_end. Can decrease due to reprocessing.
`offsets_written`	For Kafka loading, records are written if their offsets are strictly less than the value of offsets_written. For file loading, this represents the index of the most recently written file. This metric is only updated when the record/file is processed and the write is successful. Note that a successful write does not mean the record/file is durable. Can decrease due to reprocessing.
`offsets_buffered`	For Kafka loading, this represents the most recently processed record which was processed to completion and is waiting to be pushed into the Ocient Hyperscale Data Warehouse™. For file loading, this represents the index of the most recently processed file which was processed to completion and is waiting to be pushed into the Ocient® data warehouse.
`offsets_pushed`	For Kafka loading, this represents the highest offset of the batch of rows most recently pushed to the Ocient data warehouse. For file loading, this represents the highest file index of the batch of rows most recently pushed to the Ocient data warehouse. Can decrease due to reprocessing.
`offsets_end`	For Kafka loading, this represents the end offset of the partition. For file loading, this represents the file count of the partition. Should never decrease.
`offsets_durable`	For Kafka loading, records are made durable if their offsets are strictly less than the value of offsets_durable. For file loading, this represents the most durable file index. Should never decrease.
`producer_send_attempts`	Number of Kafka error topic producer send attempts
`producer_send_errors`	Number of Kafka error topic producer send errors
`bytes_pushed`	Number of bytes pushed into the Ocient data warehouse
`rows_pushed`	Number of rows pushed into the Ocient data warehouse
`pushes_attempts`	Number of attempts to push record batches into the Ocient data warehouse for this partition
`pushes_errors`	Number of attempts to push record batches into the Ocient data warehouse which resulted in error
`pushes_unacknowledged`	Number of attempts to push record batches into the Ocient data warehouse for which no response has yet been received
`invalid_state`	Number of times a code path was reached in LAT which is erroneous
`complete`	Whether or not the partition has any records left to process at the moment; this status can change often in a Kafka load but will likely not change from complete to incomplete in a file load
`errors_partition`	The number of times the LAT failed to fetch records for a particular partition
`lag`	Calculated as offsets_end - offsets_durable; In a Kafka load, lag represents the number of unprocessed records. In a file load, lag represents the number of unprocessed files. Lag becomes zero when the complete status is true.

Pipeline Metrics

The pipeline endpoint exposes aggregate metrics for the LAT instance. There are three categories of pipeline metrics:

Pipeline Metric Category	Description
`"pipeline" aggregate`	Monotonically-increasing metrics that act as lifetime counters for the running pipeline. They are independent of the current set of partitions. For instance, `rows_pushed` is the total number of rows ever pushed for this pipeline.
`"partitions" aggregate`	Metrics that depend on the current set of active partitions for the running pipeline. For instance, `total_lag` is the summed lag across all currently active partitions for the pipeline.
`pipeline-specific`	Pipeline only metrics that are not aggregates.

The Pipeline Metrics endpoint can be accessed by issuing a curl request as follows:

curl http://127.0.0.1:8080/v2/metrics/read/lat:type=pipeline

A response for this request will look as follows:

{
    "request": {
        "mbean": "lat:type=pipeline",
        "type": "read"
    },
    "value": {
        "pipeline": {
            "pushes_errors": 0,
            "pushes_attempts": 0,
            "rows_pushed": 0,
            "records_errors_column": 0,
            "records_errors_deserialization": 0,
            "records_errors_transformation": 0,
            "bytes_pushed": 0,
            "errors_partition": 0,
            "records_filter_accepted": 0,
            "records_errors_total": 0,
            "records_errors_row": 0,
            "records_filter_rejected": 0,
            "records_errors_generic": 0,
            "producer_send_attempts": 0,
            "producer_send_errors": 0
        },
        "partitions": {
            "max_lag": 0,
            "total_offsets_buffered": 0,
            "avg_lag": 0.0,
            "total_pushes_unacknowledged": 0,
            "min_lag": 0,
            "total_lag": 0,
            "total_invalid_state": 0,
            "total_offsets_pushed": 0,
            "total_records_buffered": 0,
            "total_complete": 0,
            "total_offsets_processed": 0
        },
        "paused": 0,
        "bytes_buffered": 0,
        "complete": 0,
        "workers": 32,
        "lat_version": "3.0.0"
    },
    "timestamp": 1642497956,
    "status": 200
}

Individual metrics such as lag, offsets_buffered, pushes_errors, etc. are defined within the Partition Metrics Definitions section.

However, metrics specific only to the pipeline endpoint are defined as follows:

Metric	Description
`paused`	0 if processing is active across all workers, 1 otherwise.
`bytes_buffered`	Global allocated memory in bytes.
`complete`	1 if the pipeline has finished the entire loading process, 0 otherwise.
`workers`	The number of active workers.
`lat_version`	The version of the running LAT server.

Ingest Data with Legacy LAT Reference

System Information REST Endpoints