Learned Sparse Retrieval (SPLADE) in Machine Learning

The Two Worlds Problem

Before learned sparse retrieval, the IR community was split into two camps:

1. Sparse retrieval (BM25, TF-IDF): Fast, interpretable, no training needed — but blind to semantics. The query "affordable car" would never match "budget-friendly automobile".

2. Dense retrieval (DPR, bi-encoders): Semantically powerful — but requires expensive GPU inference, opaque embeddings, and separate vector index infrastructure (FAISS, Milvus).

Hybrid approaches (combining both) helped, but they required maintaining two separate indexes and two inference paths — doubling infrastructure complexity.

The Key Insight: Neural Term Weighting

The breakthrough came from a simple question: what if we used neural networks to produce sparse vectors instead of dense ones?

A transformer model can predict, for any input text, which vocabulary terms are relevant and how important they are. If the input is "quantum physics experiments", the model might output high weights for the literal terms ("quantum", "physics", "experiments") but also for related terms that don't appear in the text ("entanglement", "Planck", "superposition"). This is neural term expansion — and it's the key to bridging the vocabulary gap while staying in sparse vector space.

The SPLADE Revolution

In 2021, Thibault Formal and colleagues at Naver Labs Europe introduced SPLADE (Sparse Lexical and Expansion Model). SPLADE takes a pre-trained BERT model, applies the MLM (masked language model) head to predict term importance across the full vocabulary, and regularizes the output with a FLOPS penalty to maintain sparsity.

The results were striking: SPLADE achieved retrieval quality within 1-2% of state-of-the-art dense retrievers on MS MARCO, while using the same inverted index infrastructure as BM25. No FAISS, no HNSW, no vector database — just a standard Lucene/Elasticsearch index with learned term weights instead of BM25 scores.

Why It Matters for Production

Learned sparse retrieval matters because it eliminates the infrastructure bifurcation of hybrid search. Instead of maintaining both an inverted index (for BM25) and a vector index (for dense retrieval), you can use a single inverted index with learned weights that captures both lexical and semantic relevance. For engineering teams at scale — whether at Naver processing Korean web search or at Indian companies building multilingual RAG systems — this operational simplification is significant.

The Smart Librarian Analogy

Remember the librarian analogy from BM25? A patron asks for books about "quantum entanglement experiments", and the librarian checks the card catalog for those exact terms.

Now imagine a smarter librarian who, upon hearing the query, thinks: "Ah, quantum entanglement — I should also check cards for 'Bell inequality', 'EPR paradox', 'superposition', and 'decoherence', because those are closely related concepts that the patron would find relevant."

That's learned sparse retrieval. The neural model acts as this smart librarian, expanding the query (or document) with semantically related terms and assigning each an importance weight.

How It Stays Sparse

The model could assign non-zero weights to every term in the vocabulary — but that would defeat the purpose (dense vectors in disguise). SPLADE uses FLOPS regularization to penalize the total number of non-zero weights:

$\\mathcal{L}_{\\text{FLOPS}} = \\sum_{j=1}^{|V|} \\bar{a}_j^2$

where $\\bar{a}_j$ is the average activation of term $j$ across the batch. This encourages the model to be selective — only activating terms that truly matter. The result is vectors with 100-300 non-zero dimensions out of 30,000+ vocabulary terms: sparse enough for efficient inverted index retrieval, but semantically enriched.

Key Insight: Learned sparse retrieval doesn't replace the inverted index — it makes it smarter. The data structure stays the same; only the term weights change from counting-based (BM25) to learned (neural).

SPLADE Formulation

Given an input text $t$ (query or document), SPLADE produces a sparse vector $\\vec{w} \\in \\mathbb{R}^{|V|}$ where $|V|$ is the vocabulary size.

Step 1: Transformer Encoding

Pass the input through a BERT-like transformer to get token-level hidden states: $\\mathbf{H} = \\text{BERT}(t) \\in \\mathbb{R}^{L \\times d}$ where $L$ is the sequence length and $d$ is the hidden dimension.

Step 2: MLM Head

Apply the masked language model head to get per-token logits over the vocabulary: $\\mathbf{Z} = \\text{MLM\\_Head}(\\mathbf{H}) \\in \\mathbb{R}^{L \\times |V|}$

Step 3: Aggregate and Sparsify

Aggregate across tokens using max-pooling, then apply log-saturation: $w_j = \\max_{i=1}^{L} \\log(1 + \\text{ReLU}(z_{i,j}))$

The $\\log(1+\\cdot)$ provides saturation (similar to BM25's term frequency saturation), and ReLU ensures non-negativity.

Step 4: Scoring

Relevance between query $q$ and document $d$ is the dot product of their sparse vectors: $\\text{score}(q, d) = \\vec{w}_q \\cdot \\vec{w}_d = \\sum_{j \\in V} w_{q,j} \\cdot w_{d,j}$

Since both vectors are sparse, this sum only involves the intersection of their non-zero terms — efficiently computed via inverted index lookup.

Training Objective

SPLADE is trained with contrastive loss plus FLOPS regularization: $\\mathcal{L} = \\mathcal{L}_{\\text{contrastive}} + \\lambda_q \\cdot \\mathcal{L}_{\\text{FLOPS}}^q + \\lambda_d \\cdot \\mathcal{L}_{\\text{FLOPS}}^d$

where: $\\mathcal{L}_{\\text{FLOPS}} = \\sum_{j=1}^{|V|} \\left(\\frac{1}{B} \\sum_{i=1}^{B} w_j^{(i)}\\right)^2$

This penalizes terms that are activated across many examples, encouraging selectivity.

DeepImpact Variant

DeepImpact (Mallia et al., SIGIR 2021) takes a simpler approach: it predicts a single importance score per existing term (no expansion), using the token's BERT embedding: $w_j = \\text{MLP}(\\mathbf{h}_j) \\quad \\text{for } j \\in \\text{terms}(d)$

This produces sparser vectors (only original terms, no expansion) but misses the semantic expansion benefit of SPLADE.

uniCOIL Variant

uniCOIL (Lin et al., 2021) uses a single linear layer on BERT token embeddings to predict term impact scores, with doc2query-T5 for document expansion before encoding.

EPIC Variant

EPIC (MacAvaney et al., 2020) predicts document term importance scores using contextual embeddings, applied as a re-weighting of existing document terms. Unlike SPLADE, EPIC does not expand the document representation but focuses on improving the quality of existing term weights through neural contextualization.

Learned sparse retrieval represents a paradigm shift in information retrieval: using neural networks to produce sparse vectors with learned term weights that combine the semantic understanding of dense models with the operational efficiency of inverted indexes. SPLADE, the leading model in this space, uses a BERT MLM head with FLOPS regularization to generate sparse representations that include neural term expansion — predicting the importance of vocabulary terms that don't even appear in the original text.

The practical impact is significant: learned sparse retrieval achieves retrieval quality within 1-2% of state-of-the-art dense retrievers while using the same inverted index infrastructure as BM25. This means organizations can upgrade from BM25 to SPLADE without deploying vector databases, FAISS clusters, or separate dense retrieval paths — a major operational simplification.

For ML engineers building RAG pipelines, learned sparse retrieval is an increasingly attractive option when you have training data and want to improve retrieval quality beyond BM25 without the infrastructure complexity of hybrid BM25+dense systems. The key tradeoff is the upfront investment in model training and document encoding versus the operational benefits of a single-index architecture.

The family of learned sparse models — SPLADE, DeepImpact, uniCOIL, EPIC — represents different points on the complexity-quality spectrum. SPLADE with distillation (SPLADE++) offers the best quality; DeepImpact and uniCOIL offer simpler training with slightly lower quality. For Indian tech companies building scalable search and RAG systems, learned sparse retrieval is particularly compelling as it leverages existing inverted index expertise and infrastructure while delivering the semantic matching quality that modern applications demand.