akaushik.org

Migrating from Fastembed ONNX to Hugging Face TEI

The specific trade-offs, the retrieval quality delta on an institutional corpus, and why the infra complexity was worth it.

Abhishek KaushikAI Engineer1 min read
The short answer
01Fastembed ONNX on CPU is a reasonable starting point. Easy to install, and you can have a working embedding pipeline in an afternoon. It stopped being the right call when retrieval quality became the constraint.
02The model catalog was the ceiling. It does not include paraphrase-multilingual-MiniLM-L12-v2 at the throughput needed on an institutional corpus, and no cross-encoder reranker like BAAI/bge-reranker-v2-m3 at all.
03Reranking matters disproportionately on this corpus. Where lexical overlap between a query and the relevant passage is low, a bi-encoder’s cosine similarity is a noisy signal. TEI adds a real reranker, at the cost of GPU nodes and k8s manifests.

Fastembed ONNX on CPU is a reasonable starting point for a RAG system. Easy to install, model catalog covers the obvious choices, and you can have a working embedding pipeline in an afternoon without standing up extra infrastructure. For VeriCite's early proof-of-concept it was the right call.

It stopped being the right call when retrieval quality became the constraint. The model catalog that is practical to serve through Fastembed doesn't include paraphrase-multilingual-MiniLM-L12-v2 at the throughput we needed on an institutional corpus, and it doesn't include a cross-encoder reranker like BAAI/bge-reranker-v2-m3 at all. Reranking matters disproportionately on the kind of corpus VeriCite deals with: documents where lexical overlap between a query and the relevant passage is low, and a bi-encoder's cosine similarity is a noisy signal.

Hugging Face TEI gives you both. Purpose-built inference server for text embeddings and rerankers, batching tuned for GPU throughput. The cost is operational: GPU nodes to run, batching configuration to manage, k8s manifests to ship for a TEI sidecar alongside Qdrant. Real infra complexity, and it doesn't pay for itself unless retrieval quality is actually on the critical path.

In this case it was. Retrieval quality on the institutional corpus improved measurably after the migration, most visibly on queries whose vocabulary diverged from the relevant document's. That is the problem the cross-encoder solves: it sees the full (query, passage) pair instead of comparing independent embeddings. Bi-encoder for the neighbourhood, reranker for the door.

Infra overhead is now a fixed cost. The model options going forward are wide open.

Common questions
Why migrate off Fastembed?

Retrieval quality became the constraint, and the catalog Fastembed can practically serve lacks the multilingual MiniLM model at the needed throughput and any cross-encoder reranker. Reranking matters on a corpus where lexical overlap between query and passage is low.

What did the migration cost?

Real infra complexity: GPU nodes to run, batching configuration to manage, and k8s manifests for a TEI sidecar alongside Qdrant. It does not pay for itself unless retrieval quality is actually on the critical path — in this case it was.

Was it worth it?

Retrieval quality on the institutional corpus improved measurably after the migration, most visibly on queries whose vocabulary diverged from the relevant document’s. Infra overhead is now a fixed cost, and the model options going forward are wide open.