bulk-cdk-core-load/io.airbyte.cdk.load.dataflow.config/AggregatePublishingConfig

AggregatePublishingConfig

data class AggregatePublishingConfig(val maxRecordsPerAgg: Long, val maxEstBytesPerAgg: Long, val stalenessDeadlinePerAgg: Duration = 5.minutes, val maxEstBytesAllAggregates: Long, val maxBufferedAggregates: Int = 5)

Configures the publishing of aggregates downstream for uploading to the destination.

By extension this controls memory usage in the destination for storing customer data. Note: the defaults presume we are storing the aggregates in memory, but if we are storing them on disk, we can handle a larger quantity of larger aggregates.

Per aggregate configuration:

Constructors

AggregatePublishingConfig

constructor(maxRecordsPerAgg: Long, maxEstBytesPerAgg: Long, stalenessDeadlinePerAgg: Duration = 5.minutes, maxEstBytesAllAggregates: Long, maxBufferedAggregates: Int = 5)

Properties

maxBufferedAggregates

val maxBufferedAggregates: Int = 5

configures the number of published aggregates we buffer before backpressuring preventing further aggregate publishes.

maxEstBytesAllAggregates

val maxEstBytesAllAggregates: Long

if the total accumulated estimated bytes for all open aggregates is greater than this value, we will publish the largest aggregate.

maxEstBytesPerAgg

val maxEstBytesPerAgg: Long

configures the size of each aggregate based on our estimated byte counts. Once an aggregate reaches this size, we publish it.

maxRecordsPerAgg

val maxRecordsPerAgg: Long

configures the max number of records in an aggregate. Once an aggregate has accumulated this many records, we publish it.

stalenessDeadlinePerAgg

val stalenessDeadlinePerAgg: Duration

is how long we will wait to flush an aggregate after it stops receiving data.