> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://ocient.ferndocs.com/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://ocient.ferndocs.com/_mcp/server.

# LAT Transform Configuration

> Configure data transformation steps in Ocient for customized data processing pipelines.

> **Warning**
>
> Data Pipelines are now the preferred method for loading data into the Ocient System. For details, see [**Load Data**](/guides/load-data).

# `transform`

A transform configuration object.

Required keys:

* Transform Configuration source only: [**transform.topics**](#transformtopics)
* File sources only: [**transform.file_groups**](#transformtopics)

## `transform.topics`

A collection of topics and their associated configuration. Keys are topic names and values are [**&#x20;topic configuration objects**](#topics) . Each `<topic>` set as a key in this object represents an Apache® Kafka® topic defined in Kafka.

> **Note**
>
> When loading from an `s3` or `local` File Source type, `transform.file_groups` should be used instead of `transform.topics`. Each key in the `file_group` must match a `file_group_name` defined in the Source section of the pipeline configuration.

| **Type:**     | object |
| ------------- | ------ |
| **Required:** | Yes    |
| **Default:**  |        |

## `transform.database`

A database to be used to fully qualify any table names that are not fully qualified.

For example, if `transform.database` is set to `myDatabase`, a table name of the form `schema.table` will become `myDatabase.schema.table`.

| **Type:**     | string |
| ------------- | ------ |
| **Required:** | No     |
| **Default:**  | null   |

## `transform.schema`

A schema to be used to fully qualify any table names that are not fully qualified. Specifying this property requires that [**transform.database**](#transformdatabase)  is specified.

For example, if `transform.database` is set to `myDatabase` and `transform.schema` is set to `mySchema`:

* A table name of the form `table` will become `myDatabase.mySchema.table`
* A table name of the form `schema.table` will become `myDatabase.schema.table`
* A table name of the form `database.schema.table` will stay as `database.schema.table`

| **Type:**     | string |
| ------------- | ------ |
| **Required:** | No     |
| **Default:**  | null   |

### Kafka Load Transform Example

```json
{
    ...
    "transform": {
        "topics": {
            "topic_1": {
                "tables": {
                    ...
                }
            },
            "topic_2": {
                "tables": {
                    ...
                }
            }
        }
    }
}
```

### File Based Load Transform Example

Unlike Kafka, File loads define file groups in the source section of the pipeline configuration. The "file_groups" defined in the source and transform sections must match.

```json
{
    "source": {
        "type": "s3",
        ...
        "file_groups": {
            "file_group_1": {
                ...
            },
            "file_group_2": {
                ...
            },
        }
    }
    "transform": {
        "file_groups": {
            "file_group_1": {
                "tables": {
                    ...
                }
            },
            "file_group_2": {
                "tables": {
                    ...
                }
            }
        }
    }
}
```

## Topics

Topic configuration objects.

Required keys:

* [**transform.topics.\<topic>.tables**](#transformtopicstopictables)

> **Note**
>
> For file based loads, `topics` are replaced by `file_groups`, but all other settings are equivalent.

### `transform.topics.<topic>.filter`

A record filter to apply at the topic level. See [**LAT Record Filtering**](/guides/lat-reference/lat-data-transformation-with-jmespath/lat-record-filtering) for details.

| **Type:**     | string |
| ------------- | ------ |
| **Required:** | Yes    |
| **Default:**  |        |

### `transform.topics.<topic>.tables`

A collection of tables and their associated configuration. Keys are table names and values are [**columns configuration**](#columns).

| **Type**:     | object |
| ------------- | ------ |
| **Required**: | Yes    |
| **Default**:  |        |

## Tables

Table configuration objects.

Required keys:

* [**transform.topics.\<topic>.tables.\<table>.columns**](#transformtopicstopictablestablecolumns)

### `transform.topics.<topic>.tables.<table>.filter`

A record filter to apply at the table level. See [**LAT Record Filtering**](/guides/lat-reference/lat-data-transformation-with-jmespath/lat-record-filtering) for details.

| **Type**:     | string |
| ------------- | ------ |
| **Required**: | Yes    |
| **Default**:  |        |

### `transform.topics.<topic>.tables.<table>.columns`

A collection of columns and their associated configurations. Keys are table names and values are [**Columns**](#columns).

## Columns

Column transformation configurations.

Required keys:

* [**transform.topics.\<topic>.tables.\<table>.columns.\<column>**](#transformtopicstopictablestablecolumnscolumn)

### `transform.topics.<topic>.tables.<table>.columns.<column>`

A column transformation keyed by a column name. A column’s value is defined as a transformation expression. The expression will query the record and return a value that is loaded into the associated column. The grammar of these expressions uses [**JMESPath**](https://jmespath.org/) enhanced with some custom Ocient® transformations and User Defined Transformations (UDTs).

| **Type**:     | string |
| ------------- | ------ |
| **Required**: | Yes    |
| **Default**:  |        |

# Complex Transform Example

```json
{
    "transform": {
        "database": "adtechdb",
        "schema": "adtech",
        "topics" : {
            "dsp.Auctions" : {
                "tables" : {
                    "auction" : {
                        "columns" : {
                            "auctionid" : "auctionId",
                            "created" : "created",
                            "bidder_nodeid" : "bidder.nodeId",
                            "deals_dealid" : "deals[].dealId"
                        }
                    },
                    "trafficSource" : {
                        "columns" : {
                            "auctionid" : "auctionId",
                            "created" : "created",
                            "sourcesellerid" : "trafficSource.sourceSellerId",
                            "domainname" : "trafficSource.domainName",
                            "contextualdataset_externalentryid" : "trafficSource.siteContextualProfile.contextualDataSet[].entries[].externalEntryId[]",
                            "dimensions_hw" : "join('x', [dimensions.height, dimensions.width])"
                        }
                    },
                    "feedback" : {
                        "columns" : {
                            "auctionid" : "auctionId",
                            "created" : "created",
                            "received" : "received",
                            "campaignid" : "EXPLODE(auctionResponseFeedbacks[].campaignId)",
                            "won" : "EXPLODE(auctionResponseFeedbacks[].won)",
                            "winningbid" : "EXPLODE(auctionResponseFeedbacks[].winningBid)",
                            "sourcelossreasoncode" : "EXPLODE(auctionResponseFeedbacks[].sourceLossReasonCode)"
                        }
                    }
                }
            }
        }
    }
}
```

## Related Links

[**Ingest Data with Legacy LAT Reference**](/guides/lat-reference/ingest-data-with-legacy-lat-reference)