RAG Pipeline
15 building blocks and models in the rag pipeline category.
Document Loader
Load documents from various sources (PDF, DOCX, web)
Text Chunker
Split documents into chunks (recursive, semantic)
Embedding Model
Generate embeddings from text
Vector Store
Store and retrieve vector embeddings
Semantic Search
Vector similarity search
Hybrid Search
Combine keyword and semantic search
Re-Ranker
Re-rank retrieved results (Cohere, BGE-reranker)
Context Assembler
Assemble context for LLM prompt
BM25 (Lexical Search)
Best Match 25 algorithm for term-frequency based document retrieval
Sparse Retrieval
Term-based retrieval using inverted indexes (TF-IDF, BM25)
Learned Sparse Retrieval (SPLADE)
Neural sparse retrieval with learned term expansion (SPLADE, DeepImpact)
ColBERT (Late Interaction)
Late interaction retrieval with per-token matching via MaxSim
Learning to Rank (LTR)
ML-based ranking combining multiple relevance signals (LambdaMART, RankNet)
Self-RAG
Self-reflective RAG with adaptive retrieval and critique tokens
LLM Generator
Language model that generates answers from retrieved context in RAG