Skip to content

Backends

The backend parameter of every component chooses where the component runs. The interface of the component stays the same whatever the backend.

Choosing a backend

backend takes a string or a backend object:

  • '<provider>' uses the default model of the provider for the task, for example 'openai'.
  • '<provider>:<model>' uses a specific model, for example 'openai:gpt-5-mini' or 'hf:HuggingFaceTB/SmolVLM-256M-Instruct'.
  • A backend object from skai.backends holds the model and the provider settings, such as the credential variable, a preconfigured client, an endpoint and extra call options.
  • None uses the default backend of the component and operation, which runs an open-source model: hf locally for most operations, and open-source models hosted by replicate for the others.
from skai.backends import DummyBackend
from skai.generation import TextToTextGenerator

instructions = 'Translate into Greek.'
by_string = TextToTextGenerator(instructions=instructions, backend='dummy:any-model').fit()
by_object = TextToTextGenerator(instructions=instructions, backend=DummyBackend(model='any-model')).fit()
print(by_string.backend_, by_object.backend_)

DummyBackend(model='any-model') DummyBackend(model='any-model')

Backend objects are scikit-learn estimators, so their parameters can be tuned like any nested parameter:

from sklearn.model_selection import GridSearchCV
from skai.backends import DummyBackend
from skai.classification import TextClassifier

X = ['good', 'bad', 'great', 'awful'] * 3
y = ['positive', 'negative'] * 6
search = GridSearchCV(TextClassifier(backend=DummyBackend()), {'backend__model': ['model-a', 'model-b']}, cv=2)
print(search.fit(X, y).best_params_)

{'backend__model': 'model-a'}

Providers

Provider Backend Runs Install Tasks
hf HFBackend locally, open-source models scikit-ai[hf] every task except talking heads, voice conversion, lip sync and fine-tuning, with diarization from scikit-ai[diarize]
openai OpenAIBackend OpenAI API scikit-ai[openai] classification, text generation, extraction, transcription, text embeddings, object detection, moments, thumbnails, speech, image generation with reference images and edits, video generation and animation, fine-tuning
litellm LiteLLMBackend Anthropic, Gemini, Mistral, Bedrock, Azure, Ollama and others through LiteLLM scikit-ai[litellm] classification, text generation, extraction, image transcription, object detection, moments, thumbnails, text and image embeddings
replicate ReplicateBackend open-source models hosted by Replicate scikit-ai[replicate] talking heads, lip sync, voice conversion, image generation, speech synthesis with voice cloning
dummy DummyBackend locally, no model included every task, with placeholder outputs for tests and documentation

A component raises UnsupportedBackendError, a ValueError, at fit when its provider is unknown or does not support its task on its input kind, and the message lists the providers that do. A missing extra raises MissingExtraError, an ImportError, that names the pip install command.

Tasks

Every operation of a component runs one backend task, and local operations run no task. The backend tasks are:

  • classify_text, classify_image, classify_audio, classify_video and classify_multimodal for classifiers.
  • generate_text for every text generator with operation='instruct'. It replaces the summarize, translate and describe_* tasks of 0.2.
  • extract_fields for every extractor, whose request carries the input kind.
  • transcribe for speech, transcribe_image for the text written in images.
  • embed_text, embed_image and embed_audio for embedders. VideoEmbedder averages embed_image over frames.
  • detect_objects, segment_image, diarize and find_moments for segmenters and reframing.
  • synthesize_speech, generate_music, generate_sound, separate_audio, enhance_audio and convert_voice for audio.
  • generate_image, edit_image, upscale_image, remove_background and pick_thumbnail for images.
  • generate_video, animate_image, animate_avatar and lip_sync for video.
  • fine_tune for adaptation='finetune'.

Backend.supported_tasks() returns the tasks a backend class implements, and Backend.supports(task, input_kind) says whether it runs a task on samples of an input kind. A prompted backend runs generate_text on audio only when its models accept audio, for example. skai.core.list_backends lists the registered providers that support a task.

from skai.backends import HFBackend, OpenAIBackend
from skai.core import list_backends

print(OpenAIBackend.supports('generate_text', 'audio'), HFBackend.supports('extract_fields', 'audio'))
print(list_backends('generate_text'), list_backends('animate_avatar'))

True True
['dummy', 'hf', 'litellm', 'openai'] ['dummy', 'replicate']

Providers and default models of every operation

A key of default_models is a task, or '<task>:<kind>' for a default that depends on the input kind, such as 'generate_text:image' for a vision model. The table lists, for every class and operation, the providers that support it and their default model, or "set model" where the provider has no default for it. Operations that no cloud API offers are marked as open-source only, and run locally on hf or on open-source models hosted by replicate. Local operations need no backend. The default column names the backend used when backend is None.

Class operation Task Default hf openai litellm replicate Notes
AudioClassifier classify classify_audio hf laion/clap-htsat-unfused gpt-audio-mini gemini/gemini-2.5-flash
ImageClassifier classify classify_image hf openai/clip-vit-base-patch32 gpt-5-mini anthropic/claude-haiku-5-5
MultimodalClassifier classify classify_multimodal hf HuggingFaceTB/SmolVLM-256M-Instruct gpt-5-mini gemini/gemini-2.5-flash
TextClassifier classify classify_text hf facebook/bart-large-mnli gpt-5-mini anthropic/claude-haiku-5-5
VideoClassifier classify classify_video hf HuggingFaceTB/SmolVLM-256M-Instruct gpt-5-mini gemini/gemini-2.5-flash
AudioExtractor extract extract_fields hf Qwen/Qwen2-Audio-7B-Instruct gpt-audio-mini gemini/gemini-2.5-flash
ImageExtractor extract extract_fields hf HuggingFaceTB/SmolVLM-256M-Instruct gpt-5-mini anthropic/claude-haiku-5-5
MultimodalExtractor extract extract_fields hf HuggingFaceTB/SmolVLM-256M-Instruct gpt-5-mini anthropic/claude-haiku-5-5
TextExtractor extract extract_fields hf Qwen/Qwen2.5-1.5B-Instruct gpt-5-mini anthropic/claude-haiku-5-5
VideoExtractor extract extract_fields hf HuggingFaceTB/SmolVLM-256M-Instruct gpt-5-mini gemini/gemini-2.5-flash
AudioEmbedder embed embed_audio hf laion/clap-htsat-unfused open-source only
ImageEmbedder embed embed_image hf clip-ViT-B-32 cohere/embed-v4.0
TextEmbedder embed embed_text hf sentence-transformers/all-MiniLM-L6-v2 text-embedding-3-small openai/text-embedding-3-small
VideoEmbedder embed embed_image hf clip-ViT-B-32 cohere/embed-v4.0
AudioSegmenter speech transcribe hf openai/whisper-small whisper-1
AudioSegmenter word transcribe hf openai/whisper-small whisper-1
AudioSegmenter speaker diarize hf pyannote/speaker-diarization-3.1 open-source only
AudioSegmenter silence local no backend
AudioSegmenter moment find_moments hf Qwen/Qwen2-Audio-7B-Instruct gpt-audio-mini gemini/gemini-2.5-flash
ImageSegmenter object detect_objects hf google/owlv2-base-patch16-ensemble gpt-5-mini anthropic/claude-haiku-5-5
ImageSegmenter mask segment_image hf facebook/sam-vit-base open-source only
TextSegmenter paragraph local no backend
TextSegmenter sentence local no backend
TextSegmenter chunk local no backend
VideoSegmenter shot local no backend
VideoSegmenter moment find_moments hf HuggingFaceTB/SmolVLM-256M-Instruct gpt-5-mini gemini/gemini-2.5-flash
AudioToAudioGenerator separate separate_audio hf HDEMUCS_HIGH_MUSDB_PLUS open-source only
AudioToAudioGenerator enhance enhance_audio hf set model open-source only
AudioToAudioGenerator convert_voice convert_voice replicate jagilley/free-vc:e4f2ff8a1d3779a2411e119dfad7d451d5f3314a8cd7003a88f88ce4c3b18d95 open-source only
AudioToAudioGenerator concatenate local no backend
AudioToTextGenerator transcribe transcribe hf openai/whisper-small whisper-1
AudioToTextGenerator instruct generate_text hf Qwen/Qwen2-Audio-7B-Instruct gpt-audio-mini gemini/gemini-2.5-flash
AudioToVideoGenerator talking_head animate_avatar replicate cjwbw/sadtalker open-source only
ImageToImageGenerator edit edit_image hf timbrooks/instruct-pix2pix gpt-image-1
ImageToImageGenerator upscale upscale_image hf caidas/swin2SR-classical-sr-x2-64 open-source only
ImageToImageGenerator remove_background remove_background hf briaai/RMBG-1.4 open-source only
ImageToImageGenerator collage local no backend
ImageToTextGenerator instruct generate_text hf HuggingFaceTB/SmolVLM-256M-Instruct gpt-5-mini anthropic/claude-haiku-5-5
ImageToTextGenerator transcribe transcribe_image hf HuggingFaceTB/SmolVLM-256M-Instruct gpt-5-mini anthropic/claude-haiku-5-5
ImageToVideoGenerator animate animate_image hf stabilityai/stable-video-diffusion-img2vid-xt sora-2
MultimodalToAudioGenerator mix local no backend
MultimodalToAudioGenerator speech synthesize_speech hf facebook/mms-tts-eng gpt-4o-mini-tts jaaari/kokoro-82m
MultimodalToImageGenerator edit edit_image hf timbrooks/instruct-pix2pix gpt-image-1
MultimodalToTextGenerator instruct generate_text hf HuggingFaceTB/SmolVLM-256M-Instruct gpt-5-mini anthropic/claude-haiku-5-5
MultimodalToVideoGenerator lip_sync lip_sync replicate bytedance/latentsync:637ce1919f807ca20da3a448ddc2743535d2853649574cd52a933120e9b9e293 open-source only
MultimodalToVideoGenerator mux local no backend
MultimodalToVideoGenerator subtitle local no backend
TextToAudioGenerator speech synthesize_speech hf facebook/mms-tts-eng gpt-4o-mini-tts jaaari/kokoro-82m
TextToAudioGenerator music generate_music hf facebook/musicgen-small open-source only
TextToAudioGenerator sound generate_sound hf cvssp/audioldm2 open-source only
TextToImageGenerator generate generate_image hf stabilityai/sd-turbo gpt-image-1 black-forest-labs/flux-schnell
TextToTextGenerator instruct generate_text hf Qwen/Qwen2.5-1.5B-Instruct gpt-5-mini anthropic/claude-haiku-5-5
TextToVideoGenerator generate generate_video hf Wan-AI/Wan2.1-T2V-1.3B-Diffusers sora-2
VideoToAudioGenerator extract local no backend
VideoToImageGenerator thumbnail pick_thumbnail hf HuggingFaceTB/SmolVLM-256M-Instruct gpt-5-mini anthropic/claude-haiku-5-5
VideoToImageGenerator frame local no backend
VideoToTextGenerator instruct generate_text hf HuggingFaceTB/SmolVLM-256M-Instruct gpt-5-mini gemini/gemini-2.5-flash
VideoToTextGenerator transcribe transcribe hf openai/whisper-small whisper-1
VideoToVideoGenerator reframe detect_objects hf google/owlv2-base-patch16-ensemble gpt-5-mini anthropic/claude-haiku-5-5
VideoToVideoGenerator concatenate local no backend

Local Hugging Face models are downloaded on first use. Replicate models should be pinned to a version, as in 'replicate:cjwbw/sadtalker:<version>', for reproducible results. ReplicateBackend clones voices with lucataco/xtts-v2 when voice is reference audio.

Prompted backends (openai, litellm, and hf for vision and chat models) score classification with one for every label the model names and zero for the others. The hf backend classifies texts, images and audio with zero-shot models that return real scores, and ignores few-shot examples. dummy returns deterministic placeholder outputs, so the same request always gives the same result.

Credentials

Cloud backends never take a secret as an argument. They take the name of the environment variable that holds it, and read it when the client is created in fit:

Backend Parameter Default variable
OpenAIBackend api_key_env OPENAI_API_KEY
ReplicateBackend api_token_env REPLICATE_API_TOKEN
LiteLLMBackend api_key_env the standard variable of each provider, such as ANTHROPIC_API_KEY

A missing variable raises MissingCredentialError naming the variable. OpenAIBackend and ReplicateBackend also take a preconfigured client, which clones of the backend share. Clients are not pickled, and are created again when a restored component is used.

Registering a backend

A backend subclasses skai.core.Backend and overrides the methods of the tasks it supports, such as generate_text or generate_image. Every method takes one request from skai.core, such as TextGenerationRequest, and returns one output. A subclass of skai.core.PromptedBackend implements complete and gets classification, text generation, extraction, image transcription, object detection, moment finding and thumbnail picking from it. skai.core.register makes the backend available by name.

from skai.core import Backend, TextGenerationRequest, register
from skai.generation import TextToTextGenerator


class FirstSentenceBackend(Backend):
    def generate_text(self, request: TextGenerationRequest) -> str:
        return request.sample.split('.')[0] + '.'


register('first-sentence', FirstSentenceBackend)
generator = TextToTextGenerator(instructions='Summarize the text.', backend='first-sentence').fit()
print(generator.predict(['It works. Everything else is detail.']))

['It works.']

Packages can also declare backends as entry points in the skai.backends group, which scikit-ai loads on first use:

[project.entry-points."skai.backends"]
my-provider = "my_package:MyBackend"