grain.transforms module

grain.transforms module#

Data transformation APIs.

List of Members#

class grain.transforms.Batch(batch_size: 'int', drop_remainder: 'bool' = False, batch_fn: 'Callable[[Sequence[Any]], Any] | None' = None)[source]#
Parameters:

  • batch_size (int)

  • drop_remainder (bool)

  • batch_fn (Callable[[Sequence[Any]], Any] | None)

class grain.transforms.Filter[source]#

Abstract base class for filter transformations for individual elements.

The pipeline will drop any element for which the filter function returns False.

Implementations should be threadsafe since they are often executed in parallel.

class grain.transforms.Map[source]#

Abstract base class for all 1:1 transformations of elements.

Implementations should be threadsafe since they are often executed in parallel.

class grain.transforms.MapWithIndex[source]#

Abstract base class for 1:1 transformations of elements and their index.

Implementations should be threadsafe since they are often executed in parallel.

class grain.transforms.RandomMap[source]#

Abstract base class for all random 1:1 transformations of elements.

Implementations should be threadsafe since they are often executed in parallel.

grain.transforms.Transformation#

alias of Batch | Map | RandomMap | TfRandomMap | Filter | FlatMap | MapWithIndex

grain.transforms.Transformations#

alias of Sequence[Batch | Map | RandomMap | TfRandomMap | Filter | FlatMap | MapWithIndex]

class grain.transforms.DatasetSelectionMap[source]#

Map from index to (constituent dataset index, index within dataset).

Note, this must be stateless, picklable and should avoid randomness to support determinism since it may be created and called concurrently in multiple processes.

Contents