RoundRobinInputPartitioner

open class RoundRobinInputPartitioner(rotateEveryNRecords: Int) : InputPartitioner

Declare a singleton of this type to have input distributed evenly across the input partitions. (The default is to ByStreamInputPartitioner.)

rotateEveryNRecords determines how often to rotate to the next partition. In testing, 10_000 seems to be the sweet spot between too much context switching and not enough load balancing.

Constructors

Link copied to clipboard
constructor(rotateEveryNRecords: Int)

Functions

Link copied to clipboard
open override fun getPartition(record: DestinationRecordRaw, numParts: Int): Int