DirectLoadTableDedupTruncateStreamLoader
class DirectLoadTableDedupTruncateStreamLoader(val stream: DestinationStream, initialStatus: DirectLoadInitialStatus, realTableName: TableName, tempTableName: TableName, columnNameMapping: ColumnNameMapping, schemaEvolutionClient: TableSchemaEvolutionClient, tableOperationsClient: TableOperationsClient, streamStateStore: StreamStateStore<DirectLoadTableExecutionConfig>, tempTableNameGenerator: TempTableNameGenerator) : StreamLoader
Stream loader implementation for deduplication + truncate mode.
This loader provides the most complex functionality, combining both deduplication and table truncation. It writes to a temporary table first, then depending on generation IDs and table status, follows different strategies:
May upsert directly to the real table if appropriate
May create a real table and upsert into it
May use a temp-temp table approach to ensure proper deduplication before overwriting the real table
This strategy ensures optimal performance while maintaining data integrity across various scenarios, including interrupted syncs and schema changes.
Constructors
Link copied to clipboard
constructor(stream: DestinationStream, initialStatus: DirectLoadInitialStatus, realTableName: TableName, tempTableName: TableName, columnNameMapping: ColumnNameMapping, schemaEvolutionClient: TableSchemaEvolutionClient, tableOperationsClient: TableOperationsClient, streamStateStore: StreamStateStore<DirectLoadTableExecutionConfig>, tempTableNameGenerator: TempTableNameGenerator)