DirectLoadTableDedupStreamLoader
class DirectLoadTableDedupStreamLoader(val stream: DestinationStream, initialStatus: DirectLoadInitialStatus, realTableName: TableName, tempTableName: TableName, columnNameMapping: ColumnNameMapping, schemaEvolutionClient: TableSchemaEvolutionClient, tableOperationsClient: TableOperationsClient, streamStateStore: StreamStateStore<DirectLoadTableExecutionConfig>) : StreamLoader
Stream loader implementation for deduplication mode.
This loader ensures uniqueness by writing to a temporary table first, then using an upsert operation to update the real table with deduplicated data. It handles cases where the temporary table may already exist from a previous run, and ensures that the schema of both tables is properly maintained.
Constructors
Link copied to clipboard
constructor(stream: DestinationStream, initialStatus: DirectLoadInitialStatus, realTableName: TableName, tempTableName: TableName, columnNameMapping: ColumnNameMapping, schemaEvolutionClient: TableSchemaEvolutionClient, tableOperationsClient: TableOperationsClient, streamStateStore: StreamStateStore<DirectLoadTableExecutionConfig>)