Segmentation

Components that split texts, images, audio and videos into labeled regions.

AudioSegmenter(*, by='speech', prompt=None, language=None, threshold=0.01, min_silence=0.3, max_vocabulary=50, use_y='auto', backend=None, batch_size=8, on_error='raise', random_state=None)

Bases: BaseSegmenter

Split every audio sample into timed regions.

Every region is a dict with label, score, start and end in seconds, and media, the audio of the region. by='speech' returns timed speech with text, by='word' timed words, by='speaker' speaker turns with speaker and text, by='silence' the parts between silences, found locally, and by='moment' the parts that match prompt.

With by='speech' or by='word', fit(X, y) with reference transcripts as y learns a vocabulary of the domain terms they contain, which the backend receives as context for spelling.

Parameters:

Name	Type	Description	Default
by	Literal['speech', 'word', 'speaker', 'silence', 'moment']	'speech', 'word', 'speaker', 'silence' or 'moment'.	'speech'
prompt	str | None	A description of the moments to find with by='moment'.	None
language	str | None	The language of the speech. It is detected when it is None.	None
threshold	float	The RMS level below which audio counts as silent with by='silence'.	0.01
min_silence	float	The shortest silence in seconds that separates two parts with by='silence'.	0.3
max_vocabulary	int	The maximum number of terms in the vocabulary, the most frequent first.	50
use_y	Literal['auto'] | bool	'auto' learns from y in fit but not in fit_transform, which pipelines use for intermediate steps. True always learns from y, and False never does.	'auto'
backend	str | Backend | None	The backend, as '<provider>', '<provider>:<model>' or a backend object. The local Hugging Face backend is used when it is None.	None
batch_size	int	The number of audio samples submitted to the backend at once.	8
on_error	Literal['raise', 'ignore']	'raise' raises when a sample fails, and 'ignore' returns None for it and warns.	'raise'
random_state	int | None	Controls the seeds sent to the backend.	None

Attributes:

Name	Type	Description
backend_	Backend | None	The resolved backend, or None with by='silence'.
vocabulary_	tuple	The domain terms learned from the reference transcripts.

Examples:

>>> import numpy as np
>>> from skai.segmentation import AudioSegmenter
>>> regions = AudioSegmenter(by='speaker', backend='dummy').fit().predict([(np.zeros(16000), 16000)])[0]
>>> [region['speaker'] for region in regions]
['SPEAKER_0', 'SPEAKER_1']

Source code in src/skai/segmentation/_audio.py

def __init__(
    self,
    *,
    by: Literal['speech', 'word', 'speaker', 'silence', 'moment'] = 'speech',
    prompt: str | None = None,
    language: str | None = None,
    threshold: float = 0.01,
    min_silence: float = 0.3,
    max_vocabulary: int = 50,
    use_y: Literal['auto'] | bool = 'auto',
    backend: str | Backend | None = None,
    batch_size: int = 8,
    on_error: Literal['raise', 'ignore'] = 'raise',
    random_state: int | None = None,
) -> None:
    self.by = by
    self.prompt = prompt
    self.language = language
    self.threshold = threshold
    self.min_silence = min_silence
    self.max_vocabulary = max_vocabulary
    self.use_y = use_y
    self.backend = backend
    self.batch_size = batch_size
    self.on_error = on_error
    self.random_state = random_state

ImageSegmenter(*, by='object', labels=None, prompt=None, backend=None, batch_size=4, on_error='raise', random_state=None)

Bases: BaseSegmenter

Find the regions of every image that show labels or match prompt, as boxes or masks.

Every region is a dict with label, score, box as (left, top, right, bottom) pixels, and media, the image cropped to the box. With by='mask' it also has mask, a grayscale image of the input size whose white pixels belong to the region, and media is transparent outside the mask.

Parameters:

Name	Type	Description	Default
by	Literal['object', 'mask']	'object' for boxes, or 'mask' for masks.	'object'
labels	list[str] | None	The kinds of regions to find.	None
prompt	str | None	A description of the regions to find, used when labels is not given.	None
backend	str | Backend | None	The backend, as '<provider>', '<provider>:<model>' or a backend object. The local Hugging Face backend is used when it is None.	None
batch_size	int	The number of images submitted to the backend at once.	4
on_error	Literal['raise', 'ignore']	'raise' raises when an image fails, and 'ignore' returns None for it and warns.	'raise'
random_state	int | None	Controls the seeds sent to the backend.	None

Attributes:

Name	Type	Description
backend_	Backend	The resolved backend.

Examples:

>>> from PIL import Image
>>> from skai.segmentation import ImageSegmenter
>>> regions = ImageSegmenter(labels=['person'], backend='dummy').fit().predict([Image.new('RGB', (64, 48))])[0]
>>> regions[0]['label'], regions[0]['media'].size
('person', (32, 24))

Source code in src/skai/segmentation/_image.py

def __init__(
    self,
    *,
    by: Literal['object', 'mask'] = 'object',
    labels: list[str] | None = None,
    prompt: str | None = None,
    backend: str | Backend | None = None,
    batch_size: int = 4,
    on_error: Literal['raise', 'ignore'] = 'raise',
    random_state: int | None = None,
) -> None:
    self.by = by
    self.labels = labels
    self.prompt = prompt
    self.backend = backend
    self.batch_size = batch_size
    self.on_error = on_error
    self.random_state = random_state

TextSegmenter(*, by='paragraph', chunk_size=1000)

Bases: BaseSegmenter

Split every text into paragraphs, sentences or chunks, locally and without a model.

Every region is a dict with label (the value of by), score (one), start and end character offsets, and media, the passage itself.

Parameters:

Name	Type	Description	Default
by	Literal['paragraph', 'sentence', 'chunk']	'paragraph' splits at blank lines, 'sentence' after sentence punctuation, and 'chunk' into pieces of chunk_size characters that join back to the text.	'paragraph'
chunk_size	int	The number of characters of a chunk.	1000

Attributes:

Name	Type	Description
backend_	None	Always None, since splitting runs locally.

Examples:

>>> from skai.segmentation import TextSegmenter
>>> regions = TextSegmenter(by='sentence').fit().predict(['One. Two.'])[0]
>>> [region['media'] for region in regions]
['One. ', 'Two.']

Source code in src/skai/segmentation/_text.py

def __init__(self, *, by: Literal['paragraph', 'sentence', 'chunk'] = 'paragraph', chunk_size: int = 1000) -> None:
    self.by = by
    self.chunk_size = chunk_size

VideoSegmenter(*, by='shot', prompt=None, threshold=0.5, n_frames=16, backend=None, batch_size=2, on_error='raise', random_state=None)

Bases: BaseSegmenter

Split every video into shots, found locally, or into the moments that match prompt.

Every region is a dict with label, score, start and end in seconds, and media, the clip of the region.

Parameters:

Name	Type	Description	Default
by	Literal['shot', 'moment']	'shot' splits at cuts, found from the color histograms of consecutive frames without a model. 'moment' asks the backend for the parts that match prompt.	'shot'
prompt	str | None	A description of the moments to find with by='moment'.	None
threshold	float	The histogram distance between consecutive frames, from 0 to 1, above which a cut is found.	0.5
n_frames	int	The number of frames sampled from every video with by='moment'.	16
backend	str | Backend | None	The backend, as '<provider>', '<provider>:<model>' or a backend object. The local Hugging Face backend is used when it is None.	None
batch_size	int	The number of videos submitted to the backend at once.	2
on_error	Literal['raise', 'ignore']	'raise' raises when a video fails, and 'ignore' returns None for it and warns.	'raise'
random_state	int | None	Controls the seeds sent to the backend.	None

Attributes:

Name	Type	Description
backend_	Backend | None	The resolved backend, or None with by='shot'.

Examples:

>>> import numpy as np
>>> from skai.core import Video
>>> from skai.segmentation import VideoSegmenter
>>> frames = np.concatenate([np.zeros((4, 8, 8, 3)), np.full((4, 8, 8, 3), 255)]).astype(np.uint8)
>>> regions = VideoSegmenter(by='shot').fit().predict([Video(frames, fps=4)])[0]
>>> [(region['start'], region['end']) for region in regions]
[(0.0, 1.0), (1.0, 2.0)]

Source code in src/skai/segmentation/_video.py

def __init__(
    self,
    *,
    by: Literal['shot', 'moment'] = 'shot',
    prompt: str | None = None,
    threshold: float = 0.5,
    n_frames: int = 16,
    backend: str | Backend | None = None,
    batch_size: int = 2,
    on_error: Literal['raise', 'ignore'] = 'raise',
    random_state: int | None = None,
) -> None:
    self.by = by
    self.prompt = prompt
    self.threshold = threshold
    self.n_frames = n_frames
    self.backend = backend
    self.batch_size = batch_size
    self.on_error = on_error
    self.random_state = random_state