> For clean Markdown of any page, append .md to the page URL. > For a complete documentation index, see https://ocient.ferndocs.com/llms.txt. > For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://ocient.ferndocs.com/_mcp/server. # Configuration Settings for Data Pipelines Data pipeline functionality enables the use of SQL statements to load data. You can use a variety of configuration settings to manage the load. See this table for the settings that you can use. | **Configuration Setting Parameter Name** | **Default** | **Data Type** | **Description** | | --------------------------------------------------------------- | ------------------------- | ------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `streamloader.extractorEngineParameters.enabled` | `true` | BOOLEAN | Whether the Loader Node attempts to bootstrap the extractor Java® process for data pipelines on start-up. If you set this parameter to `true`, the Loader Node does not become active until the Java loading process has successfully started. | | `streamloader.extractorEngineParameters.jvmMemoryConfiguration` | - `32g` for `initialHeap` | | | * `64g` for `maxHeap` * `64g` for `maxDirect` | VARCHAR | The JVM® memory configuration to use for the extractor Java process. Use strings representing data sizes in SI format: 1 KiB, 5 MiB, etc.\ You must set the three memory options in the same statement. \ \ **Example**\ `ALTER SYSTEM ALTER CONFIG SET 'streamloader.extractorEngineParameters.jvmMemoryConfiguration.initialHeap' = '32g',` `'streamloader.extractorEngineParameters.jvmMemoryConfiguration.maxHeap' = '64g',` `'streamloader.extractorEngineParameters.jvmMemoryConfiguration.maxDirect' = '64g';` | \| `streamloader.extractorEngineParameters.restPort` | 8090 | INT | Use this parameter with the `streamloader.extractorEngineParameters.portOffset` parameter to form the REST port for the extractor. (External monitoring services like Telegraf™ can query the REST server for loading process metrics.) | \| `streamloader.extractorEngineParameters.portOffset` | 0 | INT | Use this parameter with the `streamloader.extractorEngineParameters.restPort` parameter to form the REST port of the extractor. (External monitoring services like Telegraf can query the REST server for loading process metrics.) | \| `streamloader.extractorEngineParameters.configurationOption.expect.empty.file.list` | `false` | BOOLEAN | Set this value to `false` to fail a file source data pipeline that finds no files to load instead of the load completing successfully with no data loaded. | \| `streamloader.extractorEngineParameters.configurationOption.pipeline.preview.rows.limit` | 1000 | INT | The maximum number of rows a `PREVIEW PIPELINE` SQL statement can return. For example, when you use this statement, if the specified `LIMIT` value exceeds this number, then the loading process throws an error. | \| `streamloader.extractorEngineParameters.configurationOption.engine.transform.udt.jarRootDirectory` | `/opt/ocient/current/lib/extractorengine_udt` | VARCHAR | The absolute path to the directory containing JARs for data pipeline functions. To install and enable a third-party library for using data pipeline functions, install the JAR package at this location on all Loader Nodes. Then, add the chosen fully-qualified class name to the IMPORT clause of the data pipeline function. For details, see [**CREATE OR REPLACE PIPELINE FUNCTION**](/guides/sql-reference/data-definition-language-ddl-statement-reference/data-pipelines). | \| `streamloader.extractorEngineParameters.configurationOption.engine.external.jdbc.jarRootDirectory` | `/opt/ocient/current/lib/extractorengine_jdbc` | VARCHAR | The absolute path to the directory containing JARs for external source lookup functionality. For details, see [**Load Data from External Sources in Data Pipelines**](/guides/load-data/load-data-from-external-sources-in-data-pipelines). | \| `streamloader.extractorEngineParameters.configurationOption.source.record.max.size` | Dynamically calculated value at start-up based on the number of processors and amount of memory available to the JVM | BIGINT | The maximum source record size, in bytes, that the loading process tolerates before throwing an error. | \| `streamloader.extractorEngineParameters.configurationOption.s3.region` | `us-east-1` | VARCHAR | The default region for Amazon® Web Services℠ (AWS℠) S3 file sources. | \| `streamloader.extractorEngineParameters.configurationOption.s3.force.path.style.access` | `false` | BOOLEAN | Set this value to `true` to use path-style access for all S3 file sources. The default is a virtual-hosted style. | \| `streamloader.extractorEngineParameters.configurationOption.s3.force.path.style.access.if.endpoint.overriden` | `true` | BOOLEAN | Set this value to `true` to use path-style access for S3 file sources that specify endpoints. (The system still uses virtual-hosted style access for sources without endpoint overrides unless you set `streamloader.extractorEngineParameters.configurationOption.s3.force.path.style.access` to `true`). | \| `streamloader.extractorEngineParameters.configurationOption.s3.retries.count` | 10 | INT | The number of retries for the S3 file source client. | \| `streamloader.extractorEngineParameters.configurationOption.s3.netty.read.timeout.seconds` | 0 | INT | The timeout value for read operations, in seconds, for the S3 file source client. `0` means do not perform a timeout. | \| `streamloader.extractorEngineParameters.configurationOption.s3.netty.max.concurrency` | 50 | INT | Maximum number of allowed concurrent requests for the S3 file source client. | \| `streamloader.extractorEngineParameters.configurationOption.awssdk.backoff.strategy.base.delay.seconds` | 1 | INT | The base amount of time, in seconds, for calculating the time the S3 file source client waits before retrying a failed request. The range of values that the calculated time can achieve increases exponentially with each failure. You can set the maximum value using the `streamloader.extractorEngineParameters.configurationOption.awssdk.backoff.strategy.max.backoff.seconds` parameter. | \| `streamloader.extractorEngineParameters.configurationOption.awssdk.backoff.strategy.max.backoff.seconds` | 20 | INT | The maximum amount of time the S3 file source client waits before retrying a failed request. | \| `streamloader.extractorEngineParameters.configurationOption.awssdk.max.pending.connection.acquires` | 20000 | INT | The maximum number of pending connection acquires that the S3 file source client allows. | \| `streamloader.extractorEngineParameters.configurationOption.awssdk.connection.timeout.seconds` | 10 | INT | The amount of time, in seconds, that the S3 file source client waits when initially establishing a connection. | \| `streamloader.extractorEngineParameters.configurationOption.awssdk.connection.max.idle.timeout.seconds` | 300 | INT | The maximum amount of time, in seconds, that the system allows a connection established by the S3 file source client to remain open while idle. | \| `streamloader.extractorEngineParameters.configurationOption.awssdk.connection.acquisition.timeout.seconds` | 300 | INT | The amount of time, in seconds, that the S3 file source client waits when acquiring a connection from the pool. | \| `streamloader.extractorEngineParameters.configurationOption.kafka.poll.timeout.msec` | 1000 | BIGINT | The amount of time, in milliseconds, that a consumer for a pipeline with an Apache® Kafka® source waits for records to become available during a poll operation. | \| `streamloader.extractorEngineParameters.configurationOption.kafka.assign.timeout.msec` | 20000 | BIGINT | The amount of time, in milliseconds, that a `PREVIEW PIPELINE` SQL statement waits for partitions to be assigned to the consumer on a pipeline with a Kafka source. If the system does not assign any partitions when the timeout elapses, then the `PREVIEW PIPELINE` statement terminates without data processing. | \| `streamloader.extractorEngineParameters.configurationOption.kafka.activate.timeout.msec` | 5000 | BIGINT | The amount of time, in milliseconds, that a `PREVIEW PIPELINE` SQL statement waits for a partition to receive records on a pipeline with a Kafka source. | \| `streamloader.extractorEngineParameters.configurationOption.arrow.max.native.memory.usage.bytes` | 23756537856 | BIGINT | The default maximum native memory usage, in bytes, for the Apache® Parquet™ reader. | \| `streamloader.extractorEngineParameters.configurationOption.arrow.max.java.buffer.memory.usage.bytes` | 15837691904 | BIGINT | The default maximum buffer memory usage, in bytes, for the Parquet reader. | \| `streamloader.extractorEngineParameters.configurationOption.filesystem.access.directories` | `/tmp` | VARCHAR | Comma-separated list of directories where you can load server file system data.\ \ Each directory must exist.\ \ All files must be located within one of the directories.\ \ By default, you can load from the `/tmp` directory. \ | # Related Links [**Load Data**](/guides/load-data) [**Alter Default Data Pipeline Behavior**](/guides/system-administration/loading-administration/alter-default-data-pipeline-behavior)