BERTScore in Machine Learning

The N-Gram Metrics Ceiling

For decades, NLP evaluation relied on n-gram matching metrics like BLEU (Papineni et al., 2002) for machine translation and ROUGE (Lin, 2004) for summarization. The premise was simple: count how many word sequences (unigrams, bigrams, etc.) appear in both the generated text and reference text. The more overlaps, the better the quality. This worked reasonably well when translations were relatively literal and datasets were small.

But as neural models improved -- particularly after the Transformer revolution (Vaswani et al., 2017) -- a gap emerged. Models learned to generate fluent paraphrases that were semantically equivalent to references but lexically different. Consider this example from a Hindi-to-English translation task:

• Reference: "The government launched a new healthcare program."
• Candidate A: "The administration introduced a fresh medical initiative." (semantically identical)
• Candidate B: "The government healthcare program launched new." (grammatically broken)

BLEU gives Candidate A a low score (no exact word matches except "a") and Candidate B a higher score (matches "government", "launched", "new", "healthcare", "program"). This is clearly backwards. BLEU cannot distinguish synonyms ("introduced" vs "launched") or understand that word order matters semantically, not just syntactically.

Researchers in India and globally faced this issue acutely in low-resource language translation (like Tamil-to-English or Bengali-to-Hindi), where reference diversity is high and lexical overlap is naturally lower. The evaluation bottleneck was clear: we needed metrics that understand meaning, not just spelling.

The Contextual Embedding Breakthrough

The rise of contextualized language models -- ELMo (Peters et al., 2018), BERT (Devlin et al., 2019), and later RoBERTa and GPT-2 -- provided a solution. These models generate embeddings where each word's vector depends on its surrounding context. The word "bank" in "river bank" gets a different embedding than "bank" in "savings bank." This context-awareness captures semantics in a way static word vectors (Word2Vec, GloVe) never could.

The insight behind BERTScore was to leverage these contextual embeddings for evaluation. If two sentences are semantically similar, their token embeddings -- computed by BERT -- should be geometrically close in the embedding space. By matching tokens via cosine similarity instead of exact string matching, the metric could recognize paraphrases, synonyms, and reorderings as semantically valid.

The 2019 Moment: BERTScore Goes Public

Zhang et al. released the BERTScore paper on arXiv in April 2019 and presented it at ICLR 2020. The results were striking:

• On WMT machine translation datasets, BERTScore correlated 0.93 with human judgments, compared to BLEU's 0.70.
• On image captioning tasks, BERTScore showed stronger correlation than CIDEr and SPICE, the previous state-of-the-art.
• On adversarial paraphrase detection, BERTScore correctly identified semantic equivalence where BLEU failed.

The paper included careful ablations showing that model choice mattered (RoBERTa outperformed BERT, and later DeBERTa became the recommended default), IDF weighting helped in some tasks, and baseline rescaling improved interpretability by normalizing scores to a more intuitive range.

Within months, BERTScore was integrated into Hugging Face's datasets library (later spun out as evaluate), added to TorchMetrics, and adopted by researchers at Google, Meta, and academic labs worldwide. By 2026, it is cited in over 3,000 papers and is considered a standard baseline for NLG evaluation alongside BLEU, ROUGE, and METEOR.

The Coffee Shop Analogy

Imagine you ask a barista for a drink recommendation and receive two responses:

• Response A: "I suggest trying our cold brew; it has a smooth, rich flavor."
• Response B: "I recommend tasting our chilled coffee; it offers a velvety, deep taste."

To a human, these responses are nearly identical in meaning. But to a word-matching algorithm like BLEU, they share almost zero words ("our" and "it" are the only overlaps). BLEU would rate Response B as a poor match to Response A.

BERTScore solves this by understanding meaning, not memorizing spellings. It embeds each word into a high-dimensional vector space where "suggest" and "recommend" sit close together, as do "cold brew" and "chilled coffee," and "smooth, rich flavor" and "velvety, deep taste." It then matches tokens by finding the closest semantic neighbors and computes precision, recall, and F1 based on these meaning-aware alignments.

Three Key Design Decisions

1. Token-Level Matching, Not Sentence-Level: BERTScore does not compute a single sentence embedding and compare vectors (which SBERT or SentenceBERT do). Instead, it computes an embedding for every token in both the candidate and reference sentences, then performs a greedy token-by-token matching. This preserves fine-grained distinctions -- it can detect if a single word is wrong in an otherwise perfect sentence.

2. Greedy Alignment with Cosine Similarity: For each token in the reference, BERTScore finds the most similar token in the candidate (by cosine similarity of their BERT embeddings) and vice versa. This greedy approach is computationally efficient and avoids the combinatorial explosion of optimal bipartite matching. The max-similarity scores are averaged to compute precision (candidate → reference) and recall (reference → candidate), then combined into F1.

3. Model Choice as a Hyperparameter: Unlike BLEU (which is model-agnostic), BERTScore depends on the underlying language model. The original paper tested BERT-base, but later work showed that RoBERTa-large performs better, and as of 2024, microsoft/deberta-xlarge-mnli achieves the highest correlation with human evaluation. However, DeBERTa uses 2-3x more GPU memory than RoBERTa, so users must trade off quality versus computational cost. For most applications in India -- where GPU costs matter -- roberta-large is the sweet spot.

What BERTScore Does NOT Capture

While BERTScore is powerful, it has blind spots:

• Factual correctness: BERTScore measures semantic similarity, not factual accuracy. If a model hallucinates a plausible-sounding but false fact, BERTScore may still give it a high score if the hallucination is similar in style to the reference.
• Fluency and grammaticality: A word salad with high semantic overlap can score well. BERTScore does not penalize ungrammatical or disfluent text explicitly.
• Length biases: Very short or very long candidates can skew scores due to the averaging step. Recall is typically higher for longer candidates (more chances to match reference tokens), while precision is higher for shorter candidates.

Understanding these limitations is critical. BERTScore should be one signal among many -- combine it with BLEU (for lexical fidelity), ROUGE (for content coverage), perplexity (for fluency), and human evaluation (for factuality and coherence).

Mathematical Formulation

Let $x = (x_1, x_2, \ldots, x_k)$ be the candidate (generated) sentence and $\hat{x} = (\hat{x}_1, \hat{x}_2, \ldots, \hat{x}_l)$ be the reference sentence, where $k$ and $l$ are the number of tokens (after tokenization by the BERT tokenizer).

Let $\mathbf{e}(\cdot)$ denote the contextual embedding function provided by a pre-trained BERT-family model. For each token $x_i$ in the candidate, we compute its embedding $\mathbf{v}_i = \mathbf{e}(x_i)$, where $\mathbf{v}_i \in \mathbb{R}^d$ (typically $d = 768$ for BERT-base or $d = 1024$ for BERT-large). Similarly, for each token $\hat{x}_j$ in the reference, we compute $\hat{\mathbf{v}}_j = \mathbf{e}(\hat{x}_j)$.

Note on Contextual Embeddings: The embedding $\mathbf{e}(x_i)$ is not a static lookup -- it depends on the entire input sentence. BERT processes the full sentence through multiple Transformer layers, and $\mathbf{e}(x_i)$ is extracted from a specific layer (typically the last layer or a weighted combination of layers).

Cosine Similarity

The similarity between two token embeddings is measured using cosine similarity:

$\text{sim}(x_i, \hat{x}_j) = \frac{\mathbf{v}_i \cdot \hat{\mathbf{v}}_j}{\|\mathbf{v}_i\| \cdot \|\hat{\mathbf{v}}_j\|}$

Cosine similarity ranges from -1 (opposite directions) to +1 (identical directions), with 0 indicating orthogonality. In practice, BERT embeddings are typically non-negative and clustered in a cone, so cosine similarities are usually positive.

Greedy Token Matching

BERTScore uses greedy matching to align tokens:

• Recall: For each reference token $\hat{x}_j$, find the most similar candidate token:

$R_{\text{BERT}} = \frac{1}{|\hat{x}|} \sum_{\hat{x}_j \in \hat{x}} \max_{x_i \in x} \text{sim}(x_i, \hat{x}_j)$

• Precision: For each candidate token $x_i$, find the most similar reference token:

$P_{\text{BERT}} = \frac{1}{|x|} \sum_{x_i \in x} \max_{\hat{x}_j \in \hat{x}} \text{sim}(x_i, \hat{x}_j)$

• F1 Score: The harmonic mean of precision and recall:

$F_{\text{BERT}} = 2 \cdot \frac{P_{\text{BERT}} \cdot R_{\text{BERT}}}{P_{\text{BERT}} + R_{\text{BERT}}}$

In most applications, F1 is the primary metric because it balances precision and recall. However, recall is often more important for summarization (did the summary capture all key points?), while precision is more important for translation (are all generated words valid?).

Optional Enhancements

IDF Weighting: To down-weight common words ("the", "a", "is") and emphasize rare, content-bearing tokens, BERTScore optionally applies inverse document frequency (IDF) weighting:

$R_{\text{BERT}}^{\text{IDF}} = \frac{\sum_{\hat{x}_j \in \hat{x}} \text{idf}(\hat{x}_j) \cdot \max_{x_i \in x} \text{sim}(x_i, \hat{x}_j)}{\sum_{\hat{x}_j \in \hat{x}} \text{idf}(\hat{x}_j)}$

where $\text{idf}(\hat{x}_j) = \log \frac{N}{n_j}$, with $N$ being the total number of documents in a corpus and $n_j$ being the number of documents containing token $\hat{x}_j$. IDF is computed from a large reference corpus (e.g., Common Crawl or Wikipedia). The paper notes that IDF weighting is not always beneficial -- it helps in tasks like summarization where content coverage matters, but can hurt in translation where fluency words ("the", "and") are important for grammatical correctness.

Baseline Rescaling: Raw BERTScore values are difficult to interpret because they depend on the model and task. To improve readability, the authors propose baseline rescaling:

$\hat{F}_{\text{BERT}} = \frac{F_{\text{BERT}} - b}{1 - b}$

where $b$ is the average BERTScore of random sentence pairs from a large corpus. This rescales scores so that 0 represents random text and 1 represents perfect matches. The baseline $b$ is pre-computed for each language and model combination and distributed with the bert_score library.

Computational Complexity

Let $k$ be the candidate length, $l$ be the reference length, and $d$ be the embedding dimension (typically 768 or 1024).

• Embedding computation: $O((k + l) \cdot d^2 \cdot L)$ where $L$ is the number of Transformer layers (12 for BERT-base, 24 for BERT-large). This is the dominant cost and requires GPU inference.
• Greedy matching: $O(k \cdot l \cdot d)$ for computing pairwise cosine similarities, plus $O(k \cdot l)$ for finding max values.

In practice, embedding computation dominates. For a batch of 1,000 sentence pairs with average length 20 tokens, BERTScore takes ~10 seconds on an NVIDIA A100 GPU using roberta-large, compared to milliseconds for BLEU. This is why BERTScore is typically used for validation and final evaluation, not per-step training metrics.

BERTScore represents a paradigm shift in NLP evaluation, moving from lexical matching (BLEU, ROUGE) to semantic similarity via contextual embeddings. By leveraging pre-trained BERT-family models to compute token-level cosine similarities and aggregating them via greedy matching, BERTScore achieves 0.93 Pearson correlation with human judgment on machine translation tasks -- a 30% improvement over BLEU's 0.70. This makes it the de facto standard for evaluating paraphrase-heavy tasks like summarization, dialogue, and low-resource translation.

The metric's design is elegant: no training required (uses frozen BERT embeddings), supports 104 languages out-of-the-box, and provides granular precision/recall/F1 scores for debugging. Optional enhancements (IDF weighting for content coverage, baseline rescaling for interpretability) make it adaptable to diverse use cases. Mature tooling -- the official bert-score library, Hugging Face Evaluate, TorchMetrics -- ensures easy integration into research and production pipelines.

However, BERTScore is not a silver bullet. It measures semantic similarity, not truth -- hallucinations score well if fluent. It is 100x slower than BLEU (~100ms per example on GPU), making it unsuitable for real-time evaluation. BERT's 512-token limit requires manual chunking for long documents. And model choice matters critically -- scores from BERT-base are not comparable to DeBERTa-xlarge, forcing teams to standardize on a single model for the project lifecycle.

For ML engineers building chatbots in Bangalore or researchers evaluating translation at IIT Delhi, BERTScore is essential but not sufficient. Combine it with BLEU (lexical fidelity), perplexity (fluency), and human evaluation (factuality) in a multi-metric framework. Use RoBERTa-large for daily CI checks (quality-cost sweet spot), reserve DeBERTa-xlarge for final benchmarks, and always enable baseline rescaling for interpretable reporting. Instrument BERTScore in production via offline batch jobs on sampled traffic (1% of requests), log scores to dashboards (Grafana, Datadog), and alert on score drops > 2 points to catch regressions early.

BERTScore has matured from a 2019 research paper to a 2026 production-grade tool with 3,000+ citations and adoption by Hugging Face, Galileo, Comet, and countless research labs. It has redefined what 'good evaluation' means in NLG, pushing the field toward semantic understanding. As LLMs continue to generate increasingly fluent paraphrases, BERTScore's relevance will only grow -- but always remember its blind spots, and never trust a single metric in isolation.