FileChunkTask
class FileChunkTask<T>(loader: ObjectLoader, catalog: DestinationCatalog, pathFactory: ObjectStoragePathFactory, fileHandleFactory: FileHandleFactory, uploadIdGenerator: UploadIdGenerator, inputQueue: PartitionedQueue<PipelineEvent<StreamKey, DestinationRecordRaw>>, partQueue: PartitionedQueue<PipelineEvent<ObjectKey, ObjectLoaderPartFormatter.FormattedPart>>, partitioner: ObjectLoaderFormattedPartPartitioner<StreamKey, T>, partition: Int) : Task
Given an input stream of file references, reads the files and chunks them into parts, emitting those parts to the partQueue. Once finished chunking a file, it deletes it from local storage.
Constructors
Link copied to clipboard
constructor(loader: ObjectLoader, catalog: DestinationCatalog, pathFactory: ObjectStoragePathFactory, fileHandleFactory: FileHandleFactory, uploadIdGenerator: UploadIdGenerator, inputQueue: PartitionedQueue<PipelineEvent<StreamKey, DestinationRecordRaw>>, partQueue: PartitionedQueue<PipelineEvent<ObjectKey, ObjectLoaderPartFormatter.FormattedPart>>, partitioner: ObjectLoaderFormattedPartPartitioner<StreamKey, T>, partition: Int)