bulk-cdk-toolkit-load-iceberg-parquet/io.airbyte.cdk.load.toolkits.iceberg.parquet.io

Package-level declarations

Types

abstract class BaseDeltaTaskWriter : BaseTaskWriter<T>

Implementation of the Iceberg BaseTaskWriter that handles delta-based updates (insert, update, delete).

data class ChangeDescription(val change: AirbyteRecordMessageMetaChange.Change, val reason: AirbyteRecordMessageMetaChange.Reason)

ChangedValue

data class ChangedValue(val newValue: AirbyteValue, val changeDescription: ChangeDescription?)

IcebergTableCleaner

@Singleton

class IcebergTableCleaner(icebergUtil: IcebergUtil)

Removes all data from an Iceberg org.apache.iceberg.Table. This method is necessary as some catalog implementations do not clear the underlying files written to table storage.

IcebergTableWriterFactory

@Singleton

class IcebergTableWriterFactory(icebergUtil: IcebergUtil)

Factory that creates Iceberg BaseTaskWriter instances to support writing records to Iceberg data files.

The factory takes into account whether partitioning is used on the target table and whether primary keys are configured on the destination table's schema.

IcebergUtil

@Singleton

class IcebergUtil(tableIdGenerator: TableIdGenerator)

Operation

enum Operation : Enum<Operation>

Delta operations for data.

PartitionedAppendWriter

class PartitionedAppendWriter(spec: PartitionSpec, format: FileFormat, appenderFactory: FileAppenderFactory<Record>, outputFileFactory: OutputFileFactory, io: FileIO, targetFileSize: Long, schema: Schema) : PartitionedWriter<Record>

org.apache.iceberg.io.BaseTaskWriter implementation that handles append-only writes to a partitioned Iceberg org.apache.iceberg.Table.

PartitionedDeltaWriter

class PartitionedDeltaWriter(table: Table, spec: PartitionSpec, format: FileFormat, appenderFactory: FileAppenderFactory<Record>, outputFileFactory: OutputFileFactory, io: FileIO, targetFileSize: Long, schema: Schema, identifierFieldIds: Set<Int>) : BaseDeltaTaskWriter

BaseDeltaTaskWriter implementation that handles delta (insert, update, delete) writes to a partitioned Iceberg org.apache.iceberg.Table.

RecordWrapper

data class RecordWrapper(val delegate: Record, val operation: Operation) : Record

Wraps the Iceberg Record class to add the potential delta Operation associated with the record. All record operations are delegated to the provided Record object.

UnpartitionedAppendWriter

class UnpartitionedAppendWriter(spec: PartitionSpec, format: FileFormat, appenderFactory: FileAppenderFactory<Record>, outputFileFactory: OutputFileFactory, io: FileIO, targetFileSize: Long) : UnpartitionedWriter<Record>

org.apache.iceberg.io.BaseTaskWriter implementation that handles append-only writes to an un-partitioned Iceberg org.apache.iceberg.Table.

UnpartitionedDeltaWriter

class UnpartitionedDeltaWriter(table: Table, spec: PartitionSpec, format: FileFormat, appenderFactory: FileAppenderFactory<Record>, outputFileFactory: OutputFileFactory, io: FileIO, targetFileSize: Long, schema: Schema, identifierFieldIds: Set<Int>) : BaseDeltaTaskWriter

BaseDeltaTaskWriter implementation that handles delta (insert, update, delete) writes to an un-partitioned Iceberg org.apache.iceberg.Table.

Properties

AIRBYTE_CDC_DELETE_COLUMN

const val AIRBYTE_CDC_DELETE_COLUMN: String

Functions

transformValueRecursingIntoArrays

fun EnrichedAirbyteValue.transformValueRecursingIntoArrays(transformer: (AirbyteValue, AirbyteType) -> ChangedValue?)

Our Airbyte<>Iceberg schema conversion generates strongly-typed arrays.