Milvus in Machine Learning

The Gap Between ANN Libraries and Production Systems

Let us start with a common scenario. You are an ML engineer at a growing Indian e-commerce company. You have trained a great product embedding model, and FAISS gives you lightning-fast nearest neighbor search on your laptop. Life is good -- until you need to serve 50,000 queries per second across 500 million product embeddings, handle real-time updates as new products are listed, filter by category and price range, and keep the whole thing running 24/7 with replication and failover.

FAISS does not do any of that. Neither does HNSWlib or ScaNN. They are indexing libraries, not databases. They give you the engine but not the car, the transmission, the fuel system, or the seatbelts.

This is precisely the gap Milvus was designed to fill.

The Origin Story

Milvus was born at Zilliz, a company founded in 2017 by Charles Xie and a team of database researchers. The first version was released as open source in October 2019 -- well before the GenAI explosion made vector databases a household term. Early Milvus (1.x) was a monolithic system that wrapped FAISS and provided a gRPC API, persistence, and basic metadata support.

But the team quickly realized that the monolithic architecture would not scale. In 2021, they published the landmark paper "Milvus: A Purpose-Built Vector Data Management System" at ACM SIGMOD, one of the most prestigious database conferences in the world. That paper laid out the blueprint for Milvus 2.0: a fully disaggregated, cloud-native architecture with separated storage and compute, streaming and batch processing paths, and support for heterogeneous computing (CPU + GPU).

Why It Became the Default Choice at Scale

Several factors explain why Milvus became the go-to vector database for billion-scale deployments:

1. True distributed architecture: Unlike Qdrant (which added distributed mode later) or Chroma (which is primarily single-node), Milvus was designed from day one for multi-node clusters with independent scaling of query, data, and index nodes.

2. Index diversity: Milvus supports more index types than any other vector database -- IVF_FLAT, IVF_SQ8, IVF_PQ, HNSW, SCANN, DiskANN, GPU_IVF_FLAT, GPU_IVF_PQ, GPU_CAGRA, and flat brute-force. This means you can pick the exact recall-throughput-memory tradeoff for your workload.

3. GPU acceleration: Milvus was the first vector database to support GPU-accelerated indexing and search (starting with version 1.1 in 2021), and its integration with NVIDIA RAPIDS cuVS and CAGRA has pushed GPU search performance to 50x faster than CPU alternatives.

4. Mature ecosystem: Attu (GUI), VectorDBBench (benchmarking), Milvus Lite (embedded mode), and deep integrations with LangChain, LlamaIndex, and the broader AI stack.

Key Takeaway: Milvus exists because production vector search at billion scale requires a distributed database, not just an indexing library. It bridges the gap between raw ANN algorithms and the operational requirements of real-world ML systems.

Think of It as a Distributed Library for Vectors

Here is a mental model that works well: imagine a massive public library system spread across multiple buildings in a city. Each building (worker node) stores a portion of the book collection. There is a central catalog office (coordinator service) that knows which books are in which building. When you walk in and ask "find me books similar to this one," the catalog office dispatches runners to the relevant buildings, each runner searches their local shelves using an efficient indexing system (the ANN algorithm), and the results are merged and ranked before being handed back to you.

That is essentially what Milvus does with vectors. The "books" are embedding vectors, the "buildings" are shards distributed across query nodes, and the "indexing system" is one of many ANN algorithms (HNSW, IVF, DiskANN, etc.) that Milvus supports.

The Core Separation: Log and Data

The single most important architectural insight in Milvus 2.x is the separation of the log stream from the data store. Every mutation (insert, delete, upsert) first goes to a write-ahead log (originally Apache Pulsar, now also NATS or Kafka). Worker nodes consume this log to build their in-memory segments. Meanwhile, the durable data lives in object storage (MinIO, S3, Azure Blob).

Why does this matter? Because it means:

• Reads and writes are decoupled: you can scale query nodes independently of data nodes.
• Recovery is simple: if a query node crashes, a new one spins up and replays the log.
• Consistency is tunable: you get to choose between strong, bounded staleness, session, and eventual consistency depending on your tolerance for stale reads.

This is the same pattern used by systems like Apache Kafka + Flink, and it is what makes Milvus genuinely cloud-native rather than just "runs in a container."

What Milvus Does NOT Do

Milvus is not a general-purpose database. It does not replace PostgreSQL for your transactional workloads, it does not do complex SQL joins, and it does not manage your application state. It does one thing extraordinarily well: store vectors, build indices over them, and answer similarity queries at scale. Everything else -- your embedding pipeline, your re-ranker, your application logic -- lives outside Milvus.

Rule of Thumb: If your primary access pattern is "find me the k most similar items to this vector, optionally filtered by these metadata conditions," Milvus is built for you. If your primary access pattern is "join these three tables and aggregate the results," use PostgreSQL.

Mathematical Foundation

At its core, Milvus implements a distributed index over a collection $V = \{v_1, v_2, \ldots, v_n\}$ where each $v_i \in \mathbb{R}^d$ (or $\mathbb{R}^d_{\text{sparse}}$ for sparse vectors). Given a query vector $q \in \mathbb{R}^d$, a positive integer $k$, and an optional predicate $P$ over scalar metadata, Milvus returns:

$S = \underset{\substack{S' \subset V, |S'| = k \\ \forall v \in S': P(v) = \text{true}}}{\arg\min} \sum_{v \in S'} d(q, v)$

where $d(\cdot, \cdot)$ is the configured distance function.

Supported Distance Metrics

Milvus supports four primary distance/similarity functions:

• L2 (Euclidean): $d(q, v) = \sqrt{\sum_{i=1}^{d} (q_i - v_i)^2}$
• Inner Product (IP): $\text{sim}(q, v) = \sum_{i=1}^{d} q_i \cdot v_i$
• Cosine Similarity: $\text{sim}(q, v) = \frac{q \cdot v}{\|q\| \cdot \|v\|}$ (internally, Milvus normalizes vectors and uses IP)
• Jaccard (for binary vectors): $J(q, v) = \frac{|q \cap v|}{|q \cup v|}$

Index Complexity

The time complexity of a query depends on the index type:

where $n$ is the number of vectors, $d$ is the dimensionality, $M$ is the HNSW graph degree, $\text{nlist}$ is the number of IVF clusters, $\text{nprobe}$ is the number of clusters probed at query time, and $\text{ef}$ is the HNSW search beam width.

Quantization: Trading Precision for Memory

IVF_SQ8 performs scalar quantization, converting each 32-bit float to an 8-bit unsigned integer:

$v_i^{\text{quant}} = \text{round}\left(\frac{v_i - \min(V_i)}{\max(V_i) - \min(V_i)} \times 255\right)$

This reduces memory by 75% (from 4 bytes to 1 byte per dimension) while typically losing only 1-2% recall. For a corpus of 100 million 768-dimensional vectors, that is the difference between ~300 GB and ~75 GB of memory -- a savings of roughly $1,500/month (~INR 1.25 lakh/month) in cloud compute costs.

RaBitQ 1-Bit Quantization (Milvus 2.6)

Milvus 2.6 introduced RaBitQ, a more aggressive quantization that compresses each dimension to a single bit:

$v_i^{\text{binary}} = \begin{cases} 1 & \text{if } v_i > \text{median}(V_i) \\ 0 & \text{otherwise} \end{cases}$

This achieves a 72% memory reduction compared to full-precision HNSW while maintaining competitive recall through refinement passes. Combined with HNSW graph traversal, it delivers 4x throughput improvement in benchmarks.

Wrapping Up

Milvus is the most feature-rich and battle-tested open-source vector database available today. Its disaggregated, cloud-native architecture -- with separate storage (MinIO/S3), compute (query/data/index nodes), metadata (etcd), and streaming (Pulsar/NATS) layers -- enables true horizontal scaling for billion-vector workloads. With support for 10+ index types (including GPU-accelerated CAGRA), four tunable consistency levels, native hybrid search, and a smooth Lite-to-Distributed migration path, Milvus gives ML engineers unprecedented control over the recall-throughput-memory-cost tradeoff.

But that power comes with complexity. Running Milvus Distributed is not a weekend project -- it requires Kubernetes expertise, understanding of the etcd/Pulsar dependency chain, and careful capacity planning. For small-to-medium workloads (<10M vectors), simpler alternatives like Qdrant, pgvector, or even Milvus Lite may serve you better. The right choice depends on your scale, your team's operational maturity, and your budget constraints.

For Indian ML teams specifically, Milvus offers a compelling combination: open-source (no licensing costs), self-hostable (data residency compliance for BFSI and government projects), and deployable on Indian cloud regions (AWS Mumbai, Azure Central India). Start with Milvus Lite for prototyping at zero cost, move to Standalone on a single VM for early production (~INR 10,000-15,000/month), and scale to Distributed or Zilliz Cloud only when your data and traffic demand it. The pymilvus API stays the same throughout -- your code grows with your scale, not against it.