Model Serving in Machine Learning

The Gap Between Training and Value

Here's a stat that should make every data scientist uncomfortable: according to Gartner, over 85% of ML projects fail to reach production. The single biggest reason? The serving layer. Teams invest months training a model that achieves 0.92 AUC on an evaluation set, then discover that putting it behind an API that handles 10,000 requests per second with sub-100ms latency is an entirely different engineering discipline.

Training is a batch, offline process. You can afford to wait hours. Serving is an online, latency-sensitive process where every millisecond matters to user experience and revenue. A recommendation model at Flipkart that takes 500ms to respond might as well not exist -- the user has already scrolled past.

Why Can't We Just Use Flask?

This is genuinely one of the most common questions from teams deploying their first model. And the answer is: you can, for a prototype. But Flask (or FastAPI, or any generic web framework) doesn't give you:

• Dynamic batching: Accumulating individual requests into batches that exploit GPU parallelism. A batch of 32 inferences on a GPU is only marginally slower than a batch of 1, but you've served 32x the requests.
• Model versioning and hot-swapping: Loading a new model version without downtime while the old version is still serving traffic.
• GPU memory management: Efficiently sharing GPU memory across concurrent requests, especially for LLMs where the KV cache can consume gigabytes per request.
• Health checking and graceful degradation: Automatically routing traffic away from unhealthy replicas.
• Multi-model serving: Running dozens of models on the same GPU, each with different resource requirements.

Dedicated serving frameworks solve all of these problems. That's why they exist.

The LLM Inflection Point

The explosion of large language models in 2023-2025 fundamentally changed model serving. Traditional serving assumed models were relatively small (a few hundred MB), inference was fast (single-digit milliseconds), and CPU was often sufficient. LLMs shattered all three assumptions:

• A 7B-parameter model in FP16 needs ~14 GB of GPU memory just for weights.
• Autoregressive generation is inherently sequential -- each token depends on the previous one.
• The KV cache grows linearly with sequence length, creating dynamic memory pressure that traditional serving frameworks were never designed to handle.

This is why purpose-built LLM serving engines like vLLM (with PagedAttention) and NVIDIA Triton (with TensorRT-LLM backend) emerged. They solve memory management problems that generic serving frameworks simply don't address.

Key Takeaway: Model serving exists because the gap between "model works in a notebook" and "model serves millions of users reliably" requires specialized infrastructure that generic web frameworks can't provide. The LLM era has only widened this gap.

The Restaurant Kitchen Analogy

Think of model serving like running a high-volume restaurant kitchen. The trained model is your recipe. The serving infrastructure is everything else: the kitchen equipment, the line cooks, the expeditor who coordinates orders, and the system that decides how to batch similar orders together.

A single customer walks in and orders a pizza? Easy -- one cook, one oven, no coordination needed. That's your Flask prototype. But when 500 customers order simultaneously, you need an expeditor (the serving framework) who batches similar orders (dynamic batching), assigns them to available ovens (GPU scheduling), monitors cook health (health checks), and ensures no customer waits too long (latency SLOs).

The model itself is just the recipe. The serving layer is what makes the restaurant actually work.

The Two Fundamental Modes

Every serving system ultimately operates in one of two modes, and understanding this distinction is crucial:

Real-time (online) serving: The application sends a request and blocks until the response arrives. Latency matters enormously -- typically P99 < 100ms for traditional ML models, P99 < 2-5 seconds for LLM generation. This is what powers recommendation widgets, fraud detection, and chatbots.

Batch (offline) serving: Predictions are computed in bulk over a dataset, results stored for later consumption. Latency doesn't matter much -- you optimize for throughput and cost. This powers daily email recommendations, pre-computed search rankings, and periodic risk scoring.

Many production systems use both. Flipkart might pre-compute category-level recommendations in batch (cheap, done overnight) and then personalize them with a real-time model at request time (expensive, but only for the top candidates).

The Cost-Latency-Quality Triangle

Every serving decision involves three competing objectives:

• Lower latency requires more (or better) hardware, which costs more.
• Lower cost means fewer GPUs, which forces you to optimize models (quantization, distillation) at some quality loss.
• Higher quality means larger models, which need more memory and compute, increasing both latency and cost.

You cannot optimize all three simultaneously. The art of model serving is finding the point on this triangle that matches your business requirements. A Zerodha fraud detection model might sacrifice some quality for ultra-low latency. A PhonePe recommendation model might accept higher latency for better quality. An IRCTC batch scoring job might sacrifice latency entirely for minimal cost.

Formalizing Serving Performance

Let's put some math behind the intuition. A model serving system $S$ takes inference requests $r \in R$ and produces predictions $\hat{y}$ subject to performance constraints.

Latency: For a single request, the end-to-end latency is:

$L_{e2e} = L_{network} + L_{queue} + L_{preprocess} + L_{inference} + L_{postprocess}$

where $L_{inference}$ typically dominates for GPU models. For autoregressive LLMs, inference latency for generating $n$ tokens is:

$L_{LLM} = L_{prefill}(|x|) + n \cdot L_{decode}$

where $L_{prefill}$ is the time to process the input prompt of length $|x|$ (compute-bound, parallelizable) and $L_{decode}$ is the per-token generation time (memory-bandwidth-bound, sequential).

Throughput: The maximum sustainable queries per second (QPS) for a serving instance with batch size $B$ is:

$\text{QPS}_{max} = \frac{B}{L_{inference}(B)}$

Note that $L_{inference}(B)$ grows sub-linearly with $B$ on GPUs (due to parallelism), so larger batches improve throughput -- up to the point where GPU memory is exhausted.

Dynamic Batching Efficiency: Given incoming requests with arrival rate $\lambda$ and a batching window $w$, the expected batch size is:

$E[B] = \lambda \cdot w$

The batching window $w$ introduces additional latency but improves GPU utilization. The trade-off is:

$L_{total} = w + L_{inference}(E[B])$

GPU Memory Budget: For a model with $P$ parameters at precision $b$ bits, the minimum memory required is:

$M_{model} = P \times \frac{b}{8} \text{ bytes}$

For LLMs, the KV cache adds:

$M_{KV} = 2 \times n_{layers} \times d_{model} \times (|x| + |y|) \times \frac{b}{8} \times B$

where the factor 2 accounts for both keys and values, and $B$ is the number of concurrent sequences. This is often the dominant memory consumer -- a Llama 2 70B model with batch size 32 and 2K sequence length requires ~40 GB just for KV cache.

Quantization Compression Ratio: Quantizing from precision $b_1$ to $b_2$ yields:

$\text{Compression Ratio} = \frac{b_1}{b_2}$

For example, FP16 (16 bits) to INT4 (4 bits) gives a 4x compression. The quality degradation depends on the quantization method (PTQ vs. QAT) and can be measured as:

$\Delta_{quality} = \frac{|\text{metric}_{fp16} - \text{metric}_{quantized}|}{\text{metric}_{fp16}}$

Typically, well-calibrated INT8 quantization achieves $\Delta_{quality} < 0.01$ (less than 1% degradation), while INT4 may see $\Delta_{quality} \approx 0.02-0.05$ depending on the model and task.

Practical Note: In interviews and design discussions, being able to quickly estimate GPU memory requirements and throughput bounds using these formulas demonstrates a strong command of the serving space. For example: "A 7B model in FP16 needs 14 GB for weights, plus ~8 GB KV cache for batch size 16 at 2K context, so it fits on a single A100-80GB with room to spare."

Model serving is the critical infrastructure layer that transforms trained ML models into production-grade prediction services. It is both the final mile of the ML pipeline and, paradoxically, the stage that most directly determines whether an ML investment generates business value.

The serving landscape has matured rapidly from 2023 to 2026. For LLMs, vLLM with PagedAttention and continuous batching has become the dominant open-source engine, delivering 2-4x throughput improvements over naive serving. For multi-framework deployments, NVIDIA Triton (now Dynamo-Triton) remains the industry standard, used by Uber, LinkedIn, and others to serve thousands of models at scale. Managed platforms like AWS SageMaker and Google Vertex AI trade cost-efficiency for operational simplicity -- the right choice for teams without dedicated platform engineering. The key technical decisions revolve around the cost-latency-quality triangle: quantization (INT8 for nearly-free 2x gains, INT4 for aggressive 4x compression with 2-5% quality trade-off), batching strategy (dynamic for traditional ML, continuous for LLMs), and scaling policy (GPU-metric-driven HPA with asymmetric scale-up/down).

Production serving demands rigorous operational practices: canary deployments with statistical significance testing, GPU memory monitoring as a leading indicator, load shedding to prevent request queuing spirals, and pre-warmed replicas to mitigate cold start cascades. The economics are stark -- a poorly optimized serving system can cost 10x more than a well-tuned one serving the same traffic. For Indian startups, the difference between an FP16 model on an A100 (~INR 70,000/month) and an INT4 quantized model on a T4 (~INR 25,000/month) can determine product viability. Mastering model serving is not optional for any team building production ML systems.