bulk-cdk-core-load/io.airbyte.cdk.load.dataflow.state

Package-level declarations

Types

AdditionalStatsHistogram

typealias AdditionalStatsHistogram = Histogram<String>

Histogram

open class Histogram<T>(map: ConcurrentMap<T, Double> = ConcurrentHashMap())

InferredStateKeyClient

@Singleton

@Requires(property = "airbyte.destination.core.data-channel.medium", value = "STDIO")

class InferredStateKeyClient(catalog: DestinationCatalog) : StateKeyClient

Calculates state / partition keys based off per stream counters. For per-stream state, partitions are 1:1 with state messages. For global states, each state has a partition for each stream.

PartitionHistogram

typealias PartitionHistogram = Histogram<PartitionKey>

PartitionKey

data class PartitionKey(val id: String)

A record is always associated with a single partition.

SelfDescribingStateKeyClient

@Singleton

@Requires(property = "airbyte.destination.core.data-channel.medium", value = "SOCKET")

class SelfDescribingStateKeyClient : StateKeyClient

Calculates state / partition keys based off the incoming State message. States and Partitions may be either 1 to 1 or 1 to many.

StateHistogram

typealias StateHistogram = Histogram<StateKey>

StateHistogramStore

@Singleton

class StateHistogramStore

StateKey

data class StateKey(val id: Long, val partitionKeys: List<PartitionKey>) : Comparable<StateKey>

A state generally refers to a single partition but can refer to many (namely, in CDC)

StateKeyClient

interface StateKeyClient

StateReconciler

@Singleton

class StateReconciler(stateStore: StateStore, emittedStatsStore: EmittedStatsStore, consumer: OutputConsumer, @Named(value = "stateReconcilerScope") scope: CoroutineScope, @Named(value = "stateReconcilerInterval") interval: Duration?)

At an interval it checks for any complete states and flushes them to the platform.

StateStore

@Singleton

class StateStore(keyClient: StateKeyClient, histogramStore: StateHistogramStore, stateStatsEnricher: StateStatsEnricher)