bulk-cdk-core-extract/io.airbyte.cdk.read

Package-level declarations

Types

ConcurrencyResource

@Singleton

class ConcurrencyResource(maxConcurrency: Int) : Resource<ConcurrencyResource.AcquiredThread>

A Resource used to manage concurrency.

ConfiguredSyncMode

enum ConfiguredSyncMode : Enum<ConfiguredSyncMode>

ConfiguredSyncMode is equivalent to SyncMode.

CreateNoPartitions

data object CreateNoPartitions : PartitionsCreator

Feed

sealed interface Feed

Feed identifies part of the data consumed during a READ operation.

FeedBootstrap

sealed class FeedBootstrap<T : Feed>

FeedBootstrap is the input to a PartitionsCreatorFactory.

FeedReader

class FeedReader(val root: RootReader, val feed: Feed, val resourceAcquirer: ResourceAcquirer, dataChannelFormat: DataChannelFormat, val dataChannelMedium: DataChannelMedium, bufferByteSizeThresholdForFlush: Int, val clock: Clock)

A FeedReader manages the publishing of RECORD, STATE and TRACE messages for a single feed.

FieldValueChange

enum FieldValueChange : Enum<FieldValueChange>

This enum maps to AirbyteRecordMessageMetaChange.Change and AirbyteRecordMessageMetaChange.Reason enum value pairs.

Global

data class Global(val streams: List<Stream>) : Feed

Acts as a key for Airbyte STATE messages of type GLOBAL.

GlobalFeedBootstrap

class GlobalFeedBootstrap(outputConsumer: StandardOutputConsumer, metaFieldDecorator: MetaFieldDecorator, stateManager: StateManager, global: Global, dataChannelFormat: DataChannelFormat, dataChannelMedium: DataChannelMedium, bufferByteSizeThresholdForFlush: Int, clock: Clock) : FeedBootstrap<Global>

FeedBootstrap implementation for Global feeds.

PartitionReadCheckpoint

data class PartitionReadCheckpoint(val opaqueStateValue: OpaqueStateValue, val numRecords: Long, val partitionId: String? = null)

PartitionReader

interface PartitionReader

A PartitionReader, when executed via run, emits records within the corresponding partition, and completes by returning the value of the state checkpoint.

PartitionsCreator

interface PartitionsCreator

A PartitionsCreator breaks down a Feed (a stream, or some global data feed) into zero, one or more partitions. Each partition is defined and read by a PartitionReader instance. These execute concurrently, but are joined serially because the state checkpoints need to appear in sequence.

PartitionsCreatorFactory

interface PartitionsCreatorFactory

PartitionsCreatorFactory must be implemented by each source connector and serves as the entrypoint to how READ operations are executed for that connector, via the PartitionsCreator and PartitionReader instances which are ultimately created by it.

PartitionsCreatorFactorySupplier

interface PartitionsCreatorFactorySupplier<T : PartitionsCreatorFactory> : Supplier<T>

Interface to allow each toolkit to return the active PartitionsCreatorFactory class For ReadOperation to be able to iterator on active factories only.

ReadOperation

@Singleton

@Requires(property = "airbyte.connector.operation", value = "read")

@Requires(env = ["source"])

class ReadOperation(val config: SourceConfiguration, val configuredCatalog: ConfiguredAirbyteCatalog, val inputState: InputState, val stateManagerFactory: StateManagerFactory, val outputConsumer: StandardOutputConsumer, val metaFieldDecorator: MetaFieldDecorator, val resourceAcquirer: ResourceAcquirer, val partitionsCreatorFactoriesSupplier: List<PartitionsCreatorFactorySupplier<PartitionsCreatorFactory>>, @Value(value = "${airbyte.connector.data-channel.format}") val dataChannelFormat: String, @Value(value = "${airbyte.connector.data-channel.medium}") val dataChannelMedium: String, @Value(value = "${airbyte.connector.output.buffer-byte-size-threshold-for-flush:4096}") val bufferByteSizeThresholdForFlush: Int, clock: Clock) : Operation

Resource

interface Resource<T : Resource.Acquired>

Resource models a shared resource which can be acquired and released.

ResourceAcquirer

@Singleton

class ResourceAcquirer(val acquierers: List<Resource<*>>)

ResourceType

enum ResourceType : Enum<ResourceType>

RootReader

class RootReader(val stateManager: StateManager, val resourceAcquisitionHeartbeat: Duration, val timeout: Duration, val outputConsumer: StandardOutputConsumer, val metaFieldDecorator: MetaFieldDecorator, val resourceAcquirer: ResourceAcquirer, val partitionsCreatorFactories: List<PartitionsCreatorFactory>, val dataChannelFormat: DataChannelFormat, val dataChannelMedium: DataChannelMedium, val bufferByteSizeThresholdForFlush: Int, clock: Clock)

RootReader is at the heart of the READ operation. A RootReader manages multiple FeedReader instances (one per Feed) and synchronizes them using coroutines.

SocketResource

@Singleton

class SocketResource(val socketDataChannelResourceHolder: SocketDataChannelResourceHolder?) : Resource<SocketResource.AcquiredSocket>

A Resource representing a socket data channel.

StateManager

class StateManager(global: Global? = null, initialGlobalState: OpaqueStateValue? = null, initialStreamStates: Map<Stream, OpaqueStateValue?> = mapOf())

Singleton object which tracks the state of an ongoing READ operation.

StateManagerFactory

@Singleton

class StateManagerFactory(val metadataQuerierFactory: MetadataQuerier.Factory<SourceConfiguration>, val metaFieldDecorator: MetaFieldDecorator, val outputConsumer: OutputConsumer, val handler: CatalogValidationFailureHandler, @Value(value = "${airbyte.connector.data-channel.medium}") val dataChannelMedium: String, @Value(value = "${airbyte.connector.data-channel.format}") val dataChannelFormat: String)

A factory for instantiating StateManager based on the inputs of a READ. These inputs are deliberately not injected here to make testing easier.

Stream

data class Stream(val id: StreamIdentifier, val schema: Set<FieldOrMetaField>, val configuredSyncMode: ConfiguredSyncMode, val configuredPrimaryKey: List<Field>?, val configuredCursor: FieldOrMetaField?) : Feed

Acts as a key for Airbyte STATE messages of type STREAM.

StreamFeedBootstrap

class StreamFeedBootstrap(outputConsumer: StandardOutputConsumer, metaFieldDecorator: MetaFieldDecorator, stateManager: StateManager, stream: Stream, dataChannelFormat: DataChannelFormat, dataChannelMedium: DataChannelMedium, bufferByteSizeThresholdForFlush: Int, clock: Clock) : FeedBootstrap<Stream>

FeedBootstrap implementation for Stream feeds.

StreamRecordConsumer

interface StreamRecordConsumer

Emits an Airbyte RECORD message for the Stream associated with this instance.

StreamStatusManager

class StreamStatusManager(feeds: List<Feed>, emit: (AirbyteStreamStatusTraceMessage) -> Unit)

StreamStatusManager emits AirbyteStreamStatusTraceMessages in response to Feed activity events, via notifyStarting, notifyComplete and notifyFailure.

UnlimitedTimePartitionReader

interface UnlimitedTimePartitionReader : PartitionReader

A PartitionReader with no time limit for its execution.

Properties

streams

val Feed.streams: List<Stream>

List of Streams this Feed emits records for.

Functions

asProtocolSyncMode

fun ConfiguredSyncMode.asProtocolSyncMode(): SyncMode

asSyncMode

fun SyncMode.asSyncMode(): ConfiguredSyncMode

generatePartitionId

fun generatePartitionId(length: Int): String