The Sink Configuration controls the destination of data in the LAT Pipeline.

sink

A sink configuration object.

Required keys:

• sink.type

sink.type

Type of sink to use in the pipeline.

Allowed values:

• ocient: see Ocient Sink for additional configuration.
• file: see File Sink for additional configuration.

Ocient Sink

The Ocient® Sink allows LAT to connect to an Ocient cluster to write rows to one or more tables.

Required keys:

• sink.remotes

sink.remotes

Array of one or more Ocient Loader Nodes, in host:port,... format

sink.batch_records

Number of records to buffer per partition before flushing records to Ocient

sink.batch_duration

Time based flushing parameter, in milliseconds. Records will flush to Ocient after this duration has elapsed with no new activity, even if fewer than batch_records records have been processed.

sink.idle_partition_polling_period

Time based polling parameter, in milliseconds. This Sink will periodically poll the remote for progress on write durability for idle partitions.

sink.request_timeout

Request timeout when communicating with Ocient remotes, in milliseconds.

sink.request_backoff

Duration to delay after a failed request to an Ocient remote prior retrying, in milliseconds.

sink.request_jitter

Additional duration to delay after a failed request to an Ocient remote prior retrying, in milliseconds. The total delay incurred prior to a given retry is request_backoff + rand(0, request_jitter).

sink.high_watermark

High watermark memory point, in bytes. The LAT will stop pushing new rows to memory buffers. It will not resume pushing rows into the memory buffers until low_watermark is reached.

sink.low_watermark

Low watermark memory point, in bytes. After reaching high_watermark, the LAT will begin pushing rows to memory buffers again when this memory level is reached.

sink.storage_scope_id

UUID of the storage scope that rows will be associated with. The scope with the given UUID must already exist in the target cluster.

sink.skip_page_replication

A Boolean value to determine whether to omit page replicas for the specified storage scope. This is ignored if sink.storage_scope_id is not specified or has already been seen by the remotes.

sink.netty_event_loop_group_threads

The number of threads in the Netty event loop group used to communicate with remotes.

Example Ocient Sink Configuration

{
    "sink": {
        "type": "ocient",
        "remotes": ["loader0:5050", "loader1:5050", "loader2:5050"]
    }
}

File Sink

A Sink type for testing LAT pipelines that writes the transformed data to local JSONL files.

Required keys:

• sink.location

sink.location

An absolute or relative path to the location that the sink should write files to.

Example File Sink Configuration

{
    "sink": {
        "type": "file",
        "location": "out/"
    }
}

External Sink Configuration

Rather than including a sink directly within the pipeline, it is also possible to configure a pipeline to use a sink that is specified externally. Sinks can be managed (created, deleted, and more) using the LAT Client Command Line Interface. A sink must exist before a pipeline can use it.

There are three ways to configure a pipeline to use a sink.

1. If a sink is included directly within a pipeline (using the LAT Sink Configuration), it will be used.
2. If a sink is not specified within the pipeline, you can specify a sink_name that corresponds to a sink previously created using the LAT Client.
3. If neither sink nor sink_name is specified in a pipeline, the default sink will be used. If a default sink has not been created using the LAT Client, a pipeline must specify either a sink or a sink_name.

Related Links

Ingest Data with Legacy LAT Reference