How many parameters does BERT have? How does DistilBERT reduce this?

Question

123 of AI Expert Team · Accepted Answer

### **How many parameters does BERT have?**

- **BERT Base (uncased):** ~**110M** params: *Config:* 12 layers, hidden size 768, 12 heads, FFN size 3072
- **BERT Large (uncased):** ~**340M** params: *Config:* 24 layers, hidden size 1024, 16 heads, FFN size 4096

### **How does DistilBERT reduce this?**

- **Parameter count:** ~**66M** (≈ **40% smaller** than BERT Base)
- **Speed:** ≈ **60% faster** inference while retaining ≈ **97%** of BERT’s language understanding (GLUE-style) performance

### What DistilBERT changes:

**Fewer Transformer blocks**

- Cuts **layers from 12 → 6** while keeping hidden size (768) and heads (12).
- This is the main source of parameter savings (most params live in the per-layer FFN and attention projections).

**Removes components not critical at pretrain time**

- Drops **Next Sentence Prediction** (NSP) objective.
- Removes the **pooler** (and typically **segment/token-type embeddings**), shaving off extra params.

**Knowledge distillation (how it stays accurate despite being smaller)**

Trains the 6-layer **student** to imitate a 12-layer **teacher** (BERT Base) using a *triple loss*:

- **Distillation loss**: KL divergence between teacher vs student logits (with a temperature T>1, often T=2).
- **Hidden-state matching**: cosine (or L2) alignment between selected student and teacher hidden states.
- **MLM loss**: standard masked-LM cross-entropy on the true tokens.
- Result: most of BERT’s capability is “compressed” into a smaller network.

**Why the numbers work out**

- Halving layers removes roughly **half of the per-layer weights**, but embeddings + LM head remain unchanged, so the net drop is about **40%** (not 50%).
- Keeping the same hidden size/head count preserves representational capacity per layer, while distillation recovers accuracy.

### **One-liner to remember**

**BERT Base ~110M, BERT Large ~340M. DistilBERT ~66M by halving layers (12→6), removing NSP/pooler, and training with knowledge distillation—so it’s ~40% smaller and ~60% faster while keeping most of BERT’s quality.**

How many parameters does BERT have? How does DistilBERT reduce this?

TL;DR:

🫣 You've only seen 10% of the answer

Follow-Up Questions

AI & ML Theory Interview Questions

Available Topics

AI & ML Theory Interview Questions

Available Topics

How many parameters does BERT have? How does DistilBERT reduce this?

TL;DR:

🫣 You've only seen 10% of the answer

Follow-Up Questions