bulk-cdk-toolkit-load-dlq/io.airbyte.cdk.load.write.dlq/DlqLoader

DlqLoader

DeadLetterQueueLoader interface.

A DeadLetterQueueLoader is a Loader that can return records for a dead letter queue.

The Loader is expected to write the record into a destination. The batching strategy remains entirely up to the Loader implementation. To support batching, the Loader can declare its own state S.

Types

Complete

data class Complete(val rejectedRecords: List<DestinationRecordRaw>? = null) : DlqLoader.DlqLoadResult

DlqLoadResult

sealed interface DlqLoadResult

Incomplete

data object Incomplete : DlqLoader.DlqLoadResult

Functions

abstract fun accept(record: DestinationRecordRaw, state: S): DlqLoader.DlqLoadResult

Accept is the main method called for every record to process. state is the current state associated with the record. Note that in this context, state refers to the output from the most recent start call. This is not an AirbyteState.

abstract fun close()

finish

abstract fun finish(state: S): DlqLoader.Complete

Called when the data needs to be persisted. The CDK will call this if data hasn't been persisted for a given amount of time or when under memory pressure in an attempt to reclaim some memory to continue processing.

start

abstract fun start(key: StreamKey, part: Int): S

Called when starting a batch.