bulk-cdk-toolkit-load-object-storage/io.airbyte.cdk.load.write.object_storage

Package-level declarations

Types

class FilePartAccumulatorFactory(pathFactory: ObjectStoragePathFactory, outputQueue: PartitionedQueue<PipelineEvent<ObjectKey, ObjectLoaderPartFormatter.FormattedPart>>, loadStrategy: ObjectLoader)

FilePartAccumulatorLegacy

class FilePartAccumulatorLegacy(pathFactory: ObjectStoragePathFactory, stream: DestinationStream, outputQueue: PartitionedQueue<PipelineEvent<ObjectKey, ObjectLoaderPartFormatter.FormattedPart>>, loadStrategy: ObjectLoader)

ObjectLoader

interface ObjectLoader : LoadStrategy

ObjectLoader is for the use case where a destination writes records into some number of files in a file system or cloud storage provider whose client supports multipart uploads.

ObjectStorageMaxSpeedPerformanceTesterOverride

class ObjectStorageMaxSpeedPerformanceTesterOverride(client: ObjectStorageClient<*>, catalog: DestinationCatalog, objectLoader: ObjectLoader, pathFactory: PathFactory, config: ObjectStorageMaxSpeedPerformanceTesterOverrideConfig) : WriteOpOverride

ObjectStorageMaxSpeedPerformanceTesterOverrideConfig

interface ObjectStorageMaxSpeedPerformanceTesterOverrideConfig

Declare a singleton of this class to override the WriteOperation with a performance test that will ignore the source and spam an object storage provider with garbage data from memory.

ObjectStorageStreamLoader

class ObjectStorageStreamLoader<T : RemoteObject<*>>(val stream: DestinationStream, client: ObjectStorageClient<T>, destinationStateManager: DestinationStateManager<ObjectStorageDestinationState>) : StreamLoader

ObjectStorageStreamLoaderFactory

@Singleton

@Secondary

class ObjectStorageStreamLoaderFactory<T : RemoteObject<*>, U : OutputStream>(client: ObjectStorageClient<T>, destinationStateManager: DestinationStateManager<ObjectStorageDestinationState>)

Functions

metadataFor

fun DestinationConfiguration.metadataFor(stream: DestinationStream): Map<String, String>