Training and Finetuning Multi-Vector Embedding Models with Sentence Transformers

Introduction

Multi-vector embedding models have become a cornerstone of modern natural language processing (NLP) pipelines, enabling richer semantic representations by capturing multiple contextualized vectors per input text. Unlike single-vector models, such as those based on traditional sentence embeddings, multi-vector approaches offer enhanced expressiveness for tasks like retrieval, question answering, and semantic similarity. In 2026, with the proliferation of large-scale language models, the ability to train and finetune these models efficiently is more critical than ever.

This article provides a practical guide to training and finetuning multi-vector embedding models using the Sentence Transformers library. We will explore key concepts, step-by-step procedures, and best practices, illustrated with a real-world example model: Alibaba-NLP/gte-modernbert-base.

Understanding Multi-Vector Embeddings

Multi-vector embedding models, often referred to as token-level or contextual embedding models, generate a sequence of vectors—one for each token—rather than a single pooled vector. This design preserves fine-grained semantic information, making them ideal for dense retrieval and reranking tasks where subtle distinctions matter.

Key Advantages

Richer Representations: Capture word-level nuances and inter-token dependencies.
Better Retrieval Performance: Outperform single-vector models on many benchmarks, especially with late interaction techniques like ColBERT.
Flexibility: Allow pooling strategies to adapt to specific downstream tasks.

Trade-offs

Computational Cost: Higher memory and latency due to processing multiple vectors.
Complexity: Requires careful handling of variable-length sequences and pooling mechanisms.

Getting Started with Sentence Transformers

Sentence Transformers is a Python framework built on PyTorch and Transformers, designed for training and using dense embedding models. As of 2026, it supports multi-vector architectures out-of-the-box, making it accessible for both prototyping and production.

Installation

pip install sentence-transformers

Loading a Pre-Trained Model

We'll use Alibaba-NLP/gte-modernbert-base, a 0.1B parameter model optimized for sentence similarity, updated in July 2025. This model is based on the ModernBERT architecture and supports multi-vector embeddings.

from sentence_transformers import SentenceTransformer

model = SentenceTransformer('Alibaba-NLP/gte-modernbert-base')

The model outputs token-level embeddings, which can be pooled or used directly for tasks like late interaction.

Training Multi-Vector Embedding Models

Training a multi-vector model involves defining a loss function that leverages token-level outputs. Common losses include MultipleNegativesRankingLoss and CachedMultipleNegativesRankingLoss, both available in Sentence Transformers.

Data Preparation

Prepare a dataset of (query, positive, negative) triplets. In 2026, large-scale datasets like MS MARCO or custom domain-specific corpora are standard. For this example, we assume a CSV file with columns query, positive, and negative.

from datasets import load_dataset

dataset = load_dataset('csv', data_files='train.csv')

Defining the Training Setup

We'll use a contrastive objective that encourages the model to assign higher similarity to query-positive pairs than query-negative pairs.

from sentence_transformers import SentenceTransformer, losses
from sentence_transformers.trainer import SentenceTransformerTrainer
from sentence_transformers.training_args import SentenceTransformerTrainingArguments
from torch.utils.data import DataLoader

# Load the model
model = SentenceTransformer('Alibaba-NLP/gte-modernbert-base')

# Define loss (assumes the model returns token embeddings; we'll use a pooling layer)
loss = losses.MultipleNegativesRankingLoss(model=model)

# Training arguments
args = SentenceTransformerTrainingArguments(
    output_dir='./finetuned_gte',
    num_train_epochs=3,
    per_device_train_batch_size=8,
    learning_rate=2e-5,
    warmup_ratio=0.1,
    fp16=True,  # Use mixed precision for efficiency
)

# Trainer
trainer = SentenceTransformerTrainer(
    model=model, 
    args=args, 
    train_dataset=dataset['train'],
    loss=loss,
)

trainer.train()

Custom Pooling for Multi-Vector Models

If the model outputs multiple vectors, you may need to add a pooling layer to reduce them to a single vector for training. For example, using CLS pooling or mean pooling:

from sentence_transformers import models

# Add pooling to the model
transformer = models.Transformer('Alibaba-NLP/gte-modernbert-base')
pooling = models.Pooling(transformer.get_word_embedding_dimension(), pooling_mode_cls_token=True)

model = SentenceTransformer(modules=[transformer, pooling])

For late interaction models (e.g., ColBERT-style), you can keep the token embeddings and use a specialized loss like ColBERTLoss (available in recent versions).

Finetuning Strategies for 2026

As models grow and datasets evolve, finetuning requires attention to efficiency and performance.

Use LoRA for Parameter-Efficient Finetuning

Low-Rank Adaptation (LoRA) allows finetuning with minimal VRAM by training only a small set of adapters. Sentence Transformers integrates with PEFT (Parameter-Efficient Finetuning).

from peft import LoraConfig, get_peft_model

lora_config = LoraConfig(r=8, lora_alpha=32, target_modules=['q', 'v'])
model = get_peft_model(model, lora_config)

Leverage Hard Negatives

In 2026, using hard negatives (hard-to-distinguish examples) significantly improves retrieval quality. Mine them with tools like SentenceTransformers's minehardnegatives utility or a separate dense retriever.

Multi-Task Learning

Combine multiple objectives (e.g., similarity, classification, and retrieval) to make the model robust. Sentence Transformers supports multi-loss combinations via losses.CombinedLoss.

Evaluation and Deployment

After finetuning, evaluate on benchmarks like BEIR or MTEB to ensure generalization. Use sentence_transformers.evaluation to compute retrieval metrics.

For deployment, export the model to ONNX or use the Hugging Face Hub for easy serving.

model.save_pretrained('./finetuned_gte')
model.push_to_hub('my-finetuned-gte')

Conclusion

Training and finetuning multi-vector embedding models with Sentence Transformers is straightforward once you understand the token-level output and pooling strategies. By leveraging modern techniques like LoRA, hard negatives, and multi-task learning, you can achieve state-of-the-art retrieval performance efficiently. The example using Alibaba-NLP/gte-modernbert-base illustrates the process, and we encourage you to experiment with your own datasets to tailor embeddings to your domain.