bulk-cdk-core-load/io.airbyte.cdk.load.dataflow.config

Package-level declarations

Types

data class AggregatePublishingConfig(val maxRecordsPerAgg: Long, val maxEstBytesPerAgg: Long, val stalenessDeadlinePerAgg: Duration = 5.minutes, val maxEstBytesAllAggregates: Long, val maxBufferedAggregates: Int = 5)

Configures the publishing of aggregates downstream for uploading to the destination.

ConnectorInputStreams

class ConnectorInputStreams(values: List<InputStream>) : List<InputStream>

Micronaut work around - wraps the input streams to avoid injecting a List directly, and being subject to Micronaut merging like beans into a single list leading to injecting unexpected extra input streams.

DispatcherBeanFactory

@Factory

class DispatcherBeanFactory

The dispatchers (think views of thread pools) and static scopes we use for dataflow.

InputBeanFactory

@Factory

class InputBeanFactory

Conditionally creates input streams / sockets based on channel medium, then wires up a pipeline to each input with separate aggregate stores but shared state stores.