Embedding

Components that map texts, images, audio and videos to vectors.

`AudioEmbedder(*, dimensions=None, center=True, normalize=False, backend=None, batch_size=16, on_error='raise', random_state=None)`

Bases: BaseEmbedder

Map every audio sample to a fixed-size vector, learning centering and a projection from the audio in fit.

Parameters:

Name	Type	Description	Default
`dimensions`	`int \| None`	The number of dimensions of the vectors. Backends that can shorten embeddings are asked to, and other embeddings are projected with PCA learned in `fit`, or cut when `fit` gets no audio. The full embedding is kept when it is `None`.	`None`
`center`	`bool`	Whether to subtract the mean embedding of the training audio.	`True`
`normalize`	`bool`	Whether to scale every vector to unit length.	`False`
`backend`	`str \| Backend \| None`	The backend, as `'<provider>'`, `'<provider>:<model>'` or a backend object. The local Hugging Face backend is used when it is `None`.	`None`
`batch_size`	`int`	The number of audio samples submitted to the backend at once.	`16`
`on_error`	`Literal['raise', 'ignore']`	`'raise'` raises when a sample fails, and `'ignore'` returns a row of NaN for it and warns.	`'raise'`
`random_state`	`int \| None`	Controls the seeds sent to the backend.	`None`

Attributes:

Name	Type	Description
`backend_`	`Backend`	The resolved backend.
`n_features_out_`	`int`	The number of dimensions of the vectors.
`mean_`	`ndarray`	The mean embedding of the training audio, before any projection.
`components_`	`ndarray \| None`	The PCA projection to `dimensions`, or `None` when no projection is learned.

Examples:

>>> import numpy as np
>>> from skai.embedding import AudioEmbedder
>>> audio = [(np.full(1600, value), 16000) for value in (0.1, 0.2, 0.3)]
>>> AudioEmbedder(dimensions=4, backend='dummy').fit_transform(audio).shape
(3, 4)

Source code in src/skai/embedding/_audio.py

def __init__(
    self,
    *,
    dimensions: int | None = None,
    center: bool = True,
    normalize: bool = False,
    backend: str | Backend | None = None,
    batch_size: int = 16,
    on_error: Literal['raise', 'ignore'] = 'raise',
    random_state: int | None = None,
) -> None:
    self.dimensions = dimensions
    self.center = center
    self.normalize = normalize
    self.backend = backend
    self.batch_size = batch_size
    self.on_error = on_error
    self.random_state = random_state

`ImageEmbedder(*, dimensions=None, center=True, normalize=False, backend=None, batch_size=16, on_error='raise', random_state=None)`

Bases: BaseEmbedder

Map every image to a fixed-size vector, learning centering and a projection from the images in fit.

With the same joint text and image model as a TextEmbedder, for example 'hf:clip-ViT-B-32', the vectors of both embedders are comparable, so texts can retrieve images.

Parameters:

Name	Type	Description	Default
`dimensions`	`int \| None`	The number of dimensions of the vectors. Backends that can shorten embeddings are asked to, and other embeddings are projected with PCA learned in `fit`, or cut when `fit` gets no images. The full embedding is kept when it is `None`.	`None`
`center`	`bool`	Whether to subtract the mean embedding of the training images.	`True`
`normalize`	`bool`	Whether to scale every vector to unit length.	`False`
`backend`	`str \| Backend \| None`	The backend, as `'<provider>'`, `'<provider>:<model>'` or a backend object. The local Hugging Face backend is used when it is `None`.	`None`
`batch_size`	`int`	The number of images submitted to the backend at once.	`16`
`on_error`	`Literal['raise', 'ignore']`	`'raise'` raises when an image fails, and `'ignore'` returns a row of NaN for it and warns.	`'raise'`
`random_state`	`int \| None`	Controls the seeds sent to the backend.	`None`

Attributes:

Name	Type	Description
`backend_`	`Backend`	The resolved backend.
`n_features_out_`	`int`	The number of dimensions of the vectors.
`mean_`	`ndarray`	The mean embedding of the training images, before any projection.
`components_`	`ndarray \| None`	The PCA projection to `dimensions`, or `None` when no projection is learned.

Examples:

>>> from PIL import Image
>>> from skai.embedding import ImageEmbedder
>>> images = [Image.new('RGB', (8, 8), color) for color in ('red', 'green', 'blue')]
>>> ImageEmbedder(dimensions=4, backend='dummy').fit_transform(images).shape
(3, 4)

Source code in src/skai/embedding/_image.py

def __init__(
    self,
    *,
    dimensions: int | None = None,
    center: bool = True,
    normalize: bool = False,
    backend: str | Backend | None = None,
    batch_size: int = 16,
    on_error: Literal['raise', 'ignore'] = 'raise',
    random_state: int | None = None,
) -> None:
    self.dimensions = dimensions
    self.center = center
    self.normalize = normalize
    self.backend = backend
    self.batch_size = batch_size
    self.on_error = on_error
    self.random_state = random_state

`TextEmbedder(*, dimensions=None, center=True, normalize=False, backend=None, batch_size=64, on_error='raise', random_state=None)`

Bases: BaseEmbedder

Map every text to a fixed-size vector, learning centering and a projection from the texts in fit.

fit(X) embeds the texts and learns their mean, which is subtracted from every embedding when center=True. When dimensions is set and the backend cannot shorten embeddings, it also learns a PCA projection to dimensions, which keeps the directions of largest variance instead of the first entries.

Parameters:

Name	Type	Description	Default
`dimensions`	`int \| None`	The number of dimensions of the vectors. Backends that can shorten embeddings are asked to, and other embeddings are projected with PCA learned in `fit`, or cut when `fit` gets no texts. The full embedding is kept when it is `None`.	`None`
`center`	`bool`	Whether to subtract the mean embedding of the training texts.	`True`
`normalize`	`bool`	Whether to scale every vector to unit length.	`False`
`backend`	`str \| Backend \| None`	The backend, as `'<provider>'`, `'<provider>:<model>'` or a backend object. The local Hugging Face backend is used when it is `None`.	`None`
`batch_size`	`int`	The number of texts submitted to the backend at once.	`64`
`on_error`	`Literal['raise', 'ignore']`	`'raise'` raises when a text fails, and `'ignore'` returns a row of NaN for it and warns.	`'raise'`
`random_state`	`int \| None`	Controls the seeds sent to the backend.	`None`

Attributes:

Name	Type	Description
`backend_`	`Backend`	The resolved backend.
`n_features_out_`	`int`	The number of dimensions of the vectors.
`mean_`	`ndarray`	The mean embedding of the training texts, before any projection.
`components_`	`ndarray \| None`	The PCA projection to `dimensions`, or `None` when no projection is learned.

Examples:

>>> from skai.embedding import TextEmbedder
>>> embedder = TextEmbedder(dimensions=4, backend='dummy')
>>> embedder.fit_transform(['a text', 'another text']).shape
(2, 4)

Source code in src/skai/embedding/_text.py

def __init__(
    self,
    *,
    dimensions: int | None = None,
    center: bool = True,
    normalize: bool = False,
    backend: str | Backend | None = None,
    batch_size: int = 64,
    on_error: Literal['raise', 'ignore'] = 'raise',
    random_state: int | None = None,
) -> None:
    self.dimensions = dimensions
    self.center = center
    self.normalize = normalize
    self.backend = backend
    self.batch_size = batch_size
    self.on_error = on_error
    self.random_state = random_state

`VideoEmbedder(*, n_frames=8, dimensions=None, center=True, normalize=False, backend=None, batch_size=4, on_error='raise', random_state=None)`

Bases: BaseEmbedder

Map every video to the mean embedding of frames sampled evenly over it.

The frames are embedded with the image embedding model of the backend, so with a joint text and image model the vectors are comparable to those of a TextEmbedder, and texts can retrieve videos.

Parameters:

Name	Type	Description	Default
`n_frames`	`int`	The number of frames sampled from every video.	`8`
`dimensions`	`int \| None`	The number of dimensions of the vectors. Backends that can shorten embeddings are asked to, and other embeddings are projected with PCA learned in `fit`, or cut when `fit` gets no videos. The full embedding is kept when it is `None`.	`None`
`center`	`bool`	Whether to subtract the mean embedding of the training videos.	`True`
`normalize`	`bool`	Whether to scale every vector to unit length.	`False`
`backend`	`str \| Backend \| None`	The backend, as `'<provider>'`, `'<provider>:<model>'` or a backend object. The local Hugging Face backend is used when it is `None`.	`None`
`batch_size`	`int`	The number of videos submitted to the backend at once.	`4`
`on_error`	`Literal['raise', 'ignore']`	`'raise'` raises when a video fails, and `'ignore'` returns a row of NaN for it and warns.	`'raise'`
`random_state`	`int \| None`	Controls the seeds sent to the backend.	`None`

Attributes:

Name	Type	Description
`backend_`	`Backend`	The resolved backend.
`n_features_out_`	`int`	The number of dimensions of the vectors.
`mean_`	`ndarray`	The mean embedding of the training videos, before any projection.
`components_`	`ndarray \| None`	The PCA projection to `dimensions`, or `None` when no projection is learned.

Examples:

>>> import numpy as np
>>> from skai.core import Video
>>> from skai.embedding import VideoEmbedder
>>> videos = [Video(np.full((8, 8, 8, 3), value, dtype=np.uint8), fps=4) for value in (0, 100, 200)]
>>> VideoEmbedder(n_frames=4, backend='dummy').fit_transform(videos).shape
(3, 8)

Source code in src/skai/embedding/_video.py

def __init__(
    self,
    *,
    n_frames: int = 8,
    dimensions: int | None = None,
    center: bool = True,
    normalize: bool = False,
    backend: str | Backend | None = None,
    batch_size: int = 4,
    on_error: Literal['raise', 'ignore'] = 'raise',
    random_state: int | None = None,
) -> None:
    self.n_frames = n_frames
    self.dimensions = dimensions
    self.center = center
    self.normalize = normalize
    self.backend = backend
    self.batch_size = batch_size
    self.on_error = on_error
    self.random_state = random_state