Classification

Components that assign labels to texts, images, audio, videos and multimodal samples.

`AudioClassifier(*, labels=None, instructions=None, k_shot=None, adaptation='examples', backend=None, batch_size=8, on_error='raise', random_state=None)`

Bases: BaseClassifier

Assign labels to audio, such as sound events, music genres or emotions, zero-shot or few-shot.

With labels and no targets, the classifier predicts zero-shot. With targets, it learns the classes from them and passes training samples to the backend as examples, one per class by default.

Parameters:

Name	Type	Description	Default
`labels`	`list \| dict \| None`	The candidate labels, as a list, or as a dict that maps every label to a description.	`None`
`instructions`	`str \| None`	Extra guidance on how to assign the labels.	`None`
`k_shot`	`int \| ArrayLike \| None`	How many training samples are passed to the backend as examples. `None` passes one per class, `0` passes none, an integer selects that many at random, and an array of indices selects those samples.	`None`
`adaptation`	`Literal['examples', 'finetune']`	`'examples'` passes examples to the backend at prediction. `'finetune'` trains a model on all the training data in `fit`, on backends that support fine-tuning, and sends later requests to it.	`'examples'`
`backend`	`str \| Backend \| None`	The backend, as `'<provider>'`, `'<provider>:<model>'` or a backend object. The local Hugging Face backend is used when it is `None`.	`None`
`batch_size`	`int`	The number of samples submitted to the backend at once.	`8`
`on_error`	`Literal['raise', 'ignore']`	`'raise'` raises when a sample fails, and `'ignore'` returns a missing value for it and warns.	`'raise'`
`random_state`	`int \| None`	Controls the selection of examples and the seeds sent to the backend.	`None`

Attributes:

Name	Type	Description
`backend_`	`Backend`	The resolved backend.
`classes_`	`ndarray`	The class labels.
`multilabel_`	`bool`	Whether the targets are a multilabel indicator matrix.
`examples_`	`tuple`	The labeled examples passed to the backend.
`model_`	`str`	The id of the fine-tuned model. It is set only with `adaptation='finetune'`.

Examples:

>>> import numpy as np
>>> from skai.classification import AudioClassifier
>>> classifier = AudioClassifier(labels=['dog bark', 'siren'], backend='dummy').fit()
>>> classifier.predict_proba([(np.zeros(16000), 16000), (np.ones(8000), 16000)]).shape
(2, 2)

Source code in src/skai/classification/_audio.py

def __init__(
    self,
    *,
    labels: list | dict | None = None,
    instructions: str | None = None,
    k_shot: int | ArrayLike | None = None,
    adaptation: Literal['examples', 'finetune'] = 'examples',
    backend: str | Backend | None = None,
    batch_size: int = 8,
    on_error: Literal['raise', 'ignore'] = 'raise',
    random_state: int | None = None,
) -> None:
    self.labels = labels
    self.instructions = instructions
    self.k_shot = k_shot
    self.adaptation = adaptation
    self.backend = backend
    self.batch_size = batch_size
    self.on_error = on_error
    self.random_state = random_state

`ImageClassifier(*, labels=None, instructions=None, k_shot=None, adaptation='examples', backend=None, batch_size=8, on_error='raise', random_state=None)`

Bases: BaseClassifier

Assign labels to images with zero-shot or few-shot prediction.

With labels and no targets, the classifier predicts zero-shot. With targets, it learns the classes from them and passes k_shot of the training images to the backend as labeled examples.

Parameters:

Name	Type	Description	Default
`labels`	`list \| dict \| None`	The candidate labels, as a list, or as a dict that maps every label to a description.	`None`
`instructions`	`str \| None`	Extra guidance on how to assign the labels.	`None`
`k_shot`	`int \| ArrayLike \| None`	How many training images are passed to the backend as examples. `None` or `0` passes none, an integer selects that many at random, and an array of indices selects those images.	`None`
`adaptation`	`Literal['examples', 'finetune']`	`'examples'` passes examples to the backend at prediction. `'finetune'` trains a model on all the training data in `fit`, on backends that support fine-tuning, and sends later requests to it.	`'examples'`
`backend`	`str \| Backend \| None`	The backend, as `'<provider>'`, `'<provider>:<model>'` or a backend object. The local Hugging Face backend is used when it is `None`.	`None`
`batch_size`	`int`	The number of images submitted to the backend at once.	`8`
`on_error`	`Literal['raise', 'ignore']`	`'raise'` raises when an image fails, and `'ignore'` returns a missing value for it and warns.	`'raise'`
`random_state`	`int \| None`	Controls the selection of examples and the seeds sent to the backend.	`None`

Attributes:

Name	Type	Description
`backend_`	`Backend`	The resolved backend.
`classes_`	`ndarray`	The class labels.
`multilabel_`	`bool`	Whether the targets are a multilabel indicator matrix.
`examples_`	`tuple`	The labeled examples passed to the backend.
`model_`	`str`	The id of the fine-tuned model. It is set only with `adaptation='finetune'`.

Examples:

>>> from PIL import Image
>>> from skai.classification import ImageClassifier
>>> classifier = ImageClassifier(labels=['cat', 'dog'], backend='dummy').fit()
>>> classifier.predict_proba([Image.new('RGB', (32, 32))]).shape
(1, 2)

Source code in src/skai/classification/_image.py

def __init__(
    self,
    *,
    labels: list | dict | None = None,
    instructions: str | None = None,
    k_shot: int | ArrayLike | None = None,
    adaptation: Literal['examples', 'finetune'] = 'examples',
    backend: str | Backend | None = None,
    batch_size: int = 8,
    on_error: Literal['raise', 'ignore'] = 'raise',
    random_state: int | None = None,
) -> None:
    self.labels = labels
    self.instructions = instructions
    self.k_shot = k_shot
    self.adaptation = adaptation
    self.backend = backend
    self.batch_size = batch_size
    self.on_error = on_error
    self.random_state = random_state

`MultimodalClassifier(*, labels=None, instructions=None, n_frames=8, k_shot=None, adaptation='examples', backend=None, batch_size=4, on_error='raise', random_state=None)`

Bases: BaseClassifier

Assign labels to multimodal samples, zero-shot or few-shot, sending every named part in one request.

A sample is a dict, or a DataFrame row, of named texts, images, audio and videos. Strings are texts unless they are URLs or paths of existing media files. With labels and no targets, the classifier predicts zero-shot. With targets, it learns the classes from them and passes training samples to the backend as examples, one per class by default.

Parameters:

Name	Type	Description	Default
`labels`	`list \| dict \| None`	The candidate labels, as a list, or as a dict that maps every label to a description.	`None`
`instructions`	`str \| None`	Extra guidance on how to assign the labels.	`None`
`n_frames`	`int`	The number of frames sampled from every video part.	`8`
`k_shot`	`int \| ArrayLike \| None`	How many training samples are passed to the backend as examples. `None` passes one per class, `0` passes none, an integer selects that many at random, and an array of indices selects those samples.	`None`
`adaptation`	`Literal['examples', 'finetune']`	`'examples'` passes examples to the backend at prediction. `'finetune'` trains a model on all the training data in `fit`, on backends that support fine-tuning, and sends later requests to it.	`'examples'`
`backend`	`str \| Backend \| None`	The backend, as `'<provider>'`, `'<provider>:<model>'` or a backend object. The local Hugging Face backend is used when it is `None`.	`None`
`batch_size`	`int`	The number of samples submitted to the backend at once.	`4`
`on_error`	`Literal['raise', 'ignore']`	`'raise'` raises when a sample fails, and `'ignore'` returns a missing value for it and warns.	`'raise'`
`random_state`	`int \| None`	Controls the selection of examples and the seeds sent to the backend.	`None`

Attributes:

Name	Type	Description
`backend_`	`Backend`	The resolved backend.
`classes_`	`ndarray`	The class labels.
`multilabel_`	`bool`	Whether the targets are a multilabel indicator matrix.
`examples_`	`tuple`	The labeled examples passed to the backend.
`model_`	`str`	The id of the fine-tuned model. It is set only with `adaptation='finetune'`.

Examples:

>>> from PIL import Image
>>> from skai.classification import MultimodalClassifier
>>> post = {'image': Image.new('RGB', (8, 8)), 'caption': 'hello'}
>>> classifier = MultimodalClassifier(labels=['safe', 'unsafe'], backend='dummy').fit()
>>> classifier.predict([post]).shape
(1,)

Source code in src/skai/classification/_multimodal.py

def __init__(
    self,
    *,
    labels: list | dict | None = None,
    instructions: str | None = None,
    n_frames: int = 8,
    k_shot: int | ArrayLike | None = None,
    adaptation: Literal['examples', 'finetune'] = 'examples',
    backend: str | Backend | None = None,
    batch_size: int = 4,
    on_error: Literal['raise', 'ignore'] = 'raise',
    random_state: int | None = None,
) -> None:
    self.labels = labels
    self.instructions = instructions
    self.n_frames = n_frames
    self.k_shot = k_shot
    self.adaptation = adaptation
    self.backend = backend
    self.batch_size = batch_size
    self.on_error = on_error
    self.random_state = random_state

`TextClassifier(*, labels=None, instructions=None, k_shot=None, adaptation='examples', backend=None, batch_size=16, on_error='raise', random_state=None)`

Bases: BaseClassifier

Assign labels to texts with zero-shot or few-shot prediction.

With labels and no targets, the classifier predicts zero-shot. With targets, it learns the classes from them and passes k_shot of the training texts to the backend as labeled examples.

Parameters:

Name	Type	Description	Default
`labels`	`list \| dict \| None`	The candidate labels, as a list, or as a dict that maps every label to a description.	`None`
`instructions`	`str \| None`	Extra guidance on how to assign the labels.	`None`
`k_shot`	`int \| ArrayLike \| None`	How many training texts are passed to the backend as examples. `None` or `0` passes none, an integer selects that many at random, and an array of indices selects those texts.	`None`
`adaptation`	`Literal['examples', 'finetune']`	`'examples'` passes examples to the backend at prediction. `'finetune'` trains a model on all the training data in `fit`, on backends that support fine-tuning, and sends later requests to it.	`'examples'`
`backend`	`str \| Backend \| None`	The backend, as `'<provider>'`, `'<provider>:<model>'` or a backend object. The local Hugging Face backend is used when it is `None`.	`None`
`batch_size`	`int`	The number of texts submitted to the backend at once.	`16`
`on_error`	`Literal['raise', 'ignore']`	`'raise'` raises when a text fails, and `'ignore'` returns a missing value for it and warns.	`'raise'`
`random_state`	`int \| None`	Controls the selection of examples and the seeds sent to the backend.	`None`

Attributes:

Name	Type	Description
`backend_`	`Backend`	The resolved backend.
`classes_`	`ndarray`	The class labels.
`multilabel_`	`bool`	Whether the targets are a multilabel indicator matrix.
`examples_`	`tuple`	The labeled examples passed to the backend.
`model_`	`str`	The id of the fine-tuned model. It is set only with `adaptation='finetune'`.

Examples:

>>> from skai.classification import TextClassifier
>>> classifier = TextClassifier(labels=['positive', 'negative'], backend='dummy').fit()
>>> classifier.predict(['I love it.', 'I hate it.']).shape
(2,)

Source code in src/skai/classification/_text.py

def __init__(
    self,
    *,
    labels: list | dict | None = None,
    instructions: str | None = None,
    k_shot: int | ArrayLike | None = None,
    adaptation: Literal['examples', 'finetune'] = 'examples',
    backend: str | Backend | None = None,
    batch_size: int = 16,
    on_error: Literal['raise', 'ignore'] = 'raise',
    random_state: int | None = None,
) -> None:
    self.labels = labels
    self.instructions = instructions
    self.k_shot = k_shot
    self.adaptation = adaptation
    self.backend = backend
    self.batch_size = batch_size
    self.on_error = on_error
    self.random_state = random_state

`VideoClassifier(*, labels=None, instructions=None, n_frames=8, k_shot=None, adaptation='examples', backend=None, batch_size=4, on_error='raise', random_state=None)`

Bases: BaseClassifier

Assign labels to videos, zero-shot or few-shot, from frames sampled evenly over each video.

With labels and no targets, the classifier predicts zero-shot. With targets, it learns the classes from them and passes training videos to the backend as examples, one per class by default. Backends that accept audio also receive the audio track.

Parameters:

Name	Type	Description	Default
`labels`	`list \| dict \| None`	The candidate labels, as a list, or as a dict that maps every label to a description.	`None`
`instructions`	`str \| None`	Extra guidance on how to assign the labels.	`None`
`n_frames`	`int`	The number of frames sampled from every video.	`8`
`k_shot`	`int \| ArrayLike \| None`	How many training videos are passed to the backend as examples. `None` passes one per class, `0` passes none, an integer selects that many at random, and an array of indices selects those videos.	`None`
`adaptation`	`Literal['examples', 'finetune']`	`'examples'` passes examples to the backend at prediction. `'finetune'` trains a model on all the training data in `fit`, on backends that support fine-tuning, and sends later requests to it.	`'examples'`
`backend`	`str \| Backend \| None`	The backend, as `'<provider>'`, `'<provider>:<model>'` or a backend object. The local Hugging Face backend is used when it is `None`.	`None`
`batch_size`	`int`	The number of videos submitted to the backend at once.	`4`
`on_error`	`Literal['raise', 'ignore']`	`'raise'` raises when a video fails, and `'ignore'` returns a missing value for it and warns.	`'raise'`
`random_state`	`int \| None`	Controls the selection of examples and the seeds sent to the backend.	`None`

Attributes:

Name	Type	Description
`backend_`	`Backend`	The resolved backend.
`classes_`	`ndarray`	The class labels.
`multilabel_`	`bool`	Whether the targets are a multilabel indicator matrix.
`examples_`	`tuple`	The labeled examples passed to the backend.
`model_`	`str`	The id of the fine-tuned model. It is set only with `adaptation='finetune'`.

Examples:

>>> import numpy as np
>>> from skai.classification import VideoClassifier
>>> from skai.core import Video
>>> video = Video(np.zeros((20, 8, 8, 3), dtype=np.uint8), fps=10)
>>> classifier = VideoClassifier(labels=['cooking', 'sports'], n_frames=4, backend='dummy').fit()
>>> classifier.predict([video, video]).shape
(2,)

Source code in src/skai/classification/_video.py

def __init__(
    self,
    *,
    labels: list | dict | None = None,
    instructions: str | None = None,
    n_frames: int = 8,
    k_shot: int | ArrayLike | None = None,
    adaptation: Literal['examples', 'finetune'] = 'examples',
    backend: str | Backend | None = None,
    batch_size: int = 4,
    on_error: Literal['raise', 'ignore'] = 'raise',
    random_state: int | None = None,
) -> None:
    self.labels = labels
    self.instructions = instructions
    self.n_frames = n_frames
    self.k_shot = k_shot
    self.adaptation = adaptation
    self.backend = backend
    self.batch_size = batch_size
    self.on_error = on_error
    self.random_state = random_state