Hugging Face shows pooling-aware training: multi-vector retrieval stores 74.8% fewer document vectors while keeping 97.8% of its NanoBEIR score, up from 94.6%
Multi-vector models such as ColBERT retrieve well but store a lot. How training the encoder for token pooling buys better retrieval at the same storage budget.
// Key points
- Multi-vector models keep one vector per token, so a corpus of millions of documents can hold hundreds of millions of vectors; token pooling merges similar vectors to shrink storage.
- At pool_factor=4, a normally trained model retains 94.6% of its unpooled nDCG@10, and pooling-aware training raises that to 97.8%, with nDCG@10 rising from 49.03 to 51.23 while storing 74.8% fewer document vectors.
- With adaptive pooling at an average of 29.58 vectors per document (84.7% fewer), the pooling-aware model retains 97.3% versus 93.3% for the normally trained one.
- The method is implemented as a loss wrapper in Sentence Transformers, with a short reproducible training example.
Builder's takeIf AI Cloud Drive's semantic search moves to multi-vector models, storage cost is the main worry, and this post gives a reproducible trade-off. I would estimate vector counts and storage cost at pool_factor=4 first, then check recall on our own documents rather than relying on NanoBEIR.