General Worker

Model

Preprocessing

class ImageCropResizeFlatten(target_image_size)

Performs a central crop, a resize and flattening. All images are transformed into vectors of the same length, since after crop and resize operations all images are of same size.

Parameters:

target_image_size (ImageSize) – Image size to which images will be resized.

get_tf_preprocessing_fn()

Returns the function that should be used for preprocessing in TensorFlow datasets.

Returns:

TensorFlow dataset preprocessing function.

Return type:

Callable, optional

get_pt_preprocessing_fn()

Returns the function that should be used for preprocessing in PyTorch datasets.

Returns:

PyTorch dataset preprocessing function.

Return type:

Callable, optional

class ImageCropResize3Channels(required_image_size, normalize=False)

Creates 3 channels if there is only one channel, performs a central crop, a resize and optionally also normalization.

Parameters:

• required_image_size (ImageSize) – Image size to which images will be resized.

• normalize (bool) – True if normalization should be performed, False otherwise. The default value False means that normalization is not performed.

get_tf_preprocessing_fn()

Returns the function that should be used for preprocessing in TensorFlow datasets.

Returns:

TensorFlow dataset preprocessing function.

Return type:

Callable, optional

get_pt_preprocessing_fn()

Returns the function that should be used for preprocessing in PyTorch datasets.

Returns:

PyTorch dataset preprocessing function.

Return type:

Callable, optional

class TextNoOpPreprocessing

Performs no preprocessing.

get_tf_preprocessing_fn()

Returns the function that should be used for preprocessing in TensorFlow datasets.

Returns:

TensorFlow dataset preprocessing function.

Return type:

Callable, optional

get_pt_preprocessing_fn()

Returns the function that should be used for preprocessing in PyTorch datasets.

Returns:

PyTorch dataset preprocessing function.

Return type:

Callable, optional

class HFPreprocessing(name, max_length, tokenizer_params)

Preprocessing (tokenization) for the HuggingFace Transformers models (BERT and XLNet).

property needs_disabled_multithreading

Specifies whether the preprocessing should be run in a single thread.

Returns:

True if the preprocessing should not be run in multiple threads, False otherwise.

Return type:

bool

get_tf_preprocessing_fn()

Returns the function that should be used for preprocessing in TensorFlow datasets.

Returns:

TensorFlow dataset preprocessing function.

Return type:

Callable, optional

get_pt_preprocessing_fn()

Returns the function that should be used for preprocessing in PyTorch datasets.

Returns:

PyTorch dataset preprocessing function.

Return type:

Callable, optional

Reader

Classifier