Alter Default Data Pipeline Behavior

An administrator might configure some parameters that control the behavior of data pipelines to modify the default behavior across the Ocient® System or specific nodes. You can use the ALTER SYSTEM ALTER CONFIG SQL statements to control the configuration parameters.

Syntax

ALTER SYSTEM ALTER CONFIG SET 'sql.pipelineparameters.parameter_name' 'parameter_value';

Parameter	Data Type	Description
`parameter_name`	string	The name of the pipeline parameter, either general or specific to the BINARY format.
`parameter_value`	string	The value for the pipeline parameter.

General Parameters

General Parameter Name	Default	Data Type	Required Value and Description
`sql.pipelineparameters.targetTimePerExtractorTask`	600	`UINT64_T`	Target execution time in seconds for `run_extractor` tasks for file loads.
`sql.pipelineparameters.defaultPartitionsPerExtractorTask`	-1	`INT`	Number of partitions per task. If you set this parameter to -1, the number is based on available CPU cores.
`sql.pipelineparameters.defaultExtractorCores`	-1	`INT`	Number of cores to use in a task. If you set this parameter to -1, the number is based on available CPU cores.

Binary Format Parameters

You can modify these parameters to change the default behavior of the BINARY data format extraction.

Binary Format Parameter Name	Default	Data Type	Required Value and Description
`sql.pipelineparameters.extract.binary.defaultEndianness`	`big`	string	The endianness of transforms when you do not specify it in another way. Supported values are `big` or `little`.
`sql.pipelineparameters.extract.binary.defaultCharset`	ibm1047	string	Determines the default character set to use when you load data in the `BINARY` format.
`sql.pipelineparameters.extract.binary.defaultAutoTrimPadding`	true	Boolean	Determines the default for whether a padding character should be trimmed when decoding data in the `BINARY` format to a string.
`sql.pipelineparameters.extract.binary.defaultPaddingCharacter`	’ ‘	`CHAR`	Determines the default padding character to trim from the end of a string when decoding data in the `BINARY` format. This parameter is effective only if the `extract.binary.defaultAutoTrimPadding` parameter is set to `true`.

Examples

Set the target execution time for the extractor task to 600 seconds.

ALTER SYSTEM ALTER CONFIG SET 'sql.pipelineparameters.targetTimePerExtractorTask' 600;

Set the default endianness of transforms to little endian for the BINARY extract.

ALTER SYSTEM ALTER CONFIG SET 'sql.pipelineparameters.extract.binary.defaultEndianness' 'little';

The configuration values only go into effect after the ALTER SQL statement succeeds and you restart the affected nodes.

Load Data

Data Types for Data Pipelines

Data Formats for Data Pipelines

Configuration Settings for Data Pipelines