๐ง Embeddings Fundamentals
๐ฏ What are Word Embeddings?
Word embeddings are dense, low-dimensional vector representations of words that capture semantic meaning. Instead of treating words as discrete symbols, embeddings map them to points in a continuous vector space where similar words are close together. They're the foundation of modern NLP.
Why Embeddings Matter
Semantic similarity: Similar words โ similar vectors. Dimensionality reduction: 50K words โ 300 dims. Generalization: Transfer knowledge across tasks. Foundation: Basis for all modern NLP. Efficient: Dense vectors, compact. Learnable: Can be trained or fine-tuned.
๐ Dense vectors๐ Semantic meaning๐ Continuous space๐ Foundation NLP
# Word embeddings โ the big picture
import numpy as np
# FROM DISCRETE TO CONTINUOUS
print("=== Discrete vs Continuous Representations ===")
print("""
Discrete (One-Hot):
- Vocabulary: 50,000 words
- Each word: 50,000-dim vector
- "cat" = [0, 0, ..., 1, ..., 0]
- Sparse, no similarity
- No semantic meaning
Continuous (Embeddings):
- Vocabulary: 50,000 words
- Each word: 300-dim vector
- "cat" = [0.2, -0.5, 0.8, ..., 0.1]
- Dense, captures similarity
- "cat" โ "kitten"
- Semantic relationships
""")
# SIMPLE EXAMPLE
print("=== Simple Example ===")
print("""
Vocabulary: {cat, dog, kitten, puppy, car, truck}
Without embeddings:
cat โ dog โ kitten (all equally different)
With embeddings:
cat โ kitten (similar vectors)
dog โ puppy (similar vectors)
cat โ dog (both animals)
car โ truck (both vehicles)
cat โ car (different categories)
Vector space:
- Animals cluster together
- Vehicles cluster together
- Within clusters, fine-grained similarity
""")
# DIMENSIONALITY
print("\n=== Dimensionality ===")
print("""
Popular dimensions:
- 50-100: Small, fast, less expressive
- 100-300: Standard (Word2Vec, GloVe)
- 300-500: Rich, larger models
- 768: BERT-base
- 1024: BERT-large
- 4096: GPT-3
Trade-offs:
Smaller dims:
โ Faster
โ Less memory
โ Less expressive
โ Miss subtle meanings
Larger dims:
โ More expressive
โ Better performance
โ Slower
โ More memory
โ Harder to train
Rule of thumb:
- 100-300 for most tasks
- Larger for complex language understanding
- Depends on vocabulary size
""")
# USE CASES
print("\n=== Common Use Cases ===")
print("""
1. TEXT CLASSIFICATION
- Sentiment analysis
- Topic classification
- Spam detection
2. SIMILARITY
- Document similarity
- Search & retrieval
- Recommendation
3. NAMED ENTITY RECOGNITION
- Feature input for NER
- Entity linking
- Relation extraction
4. MACHINE TRANSLATION
- Cross-lingual embeddings
- Word alignment
- Neural MT input
5. QUESTION ANSWERING
- Semantic search
- Answer ranking
- Context matching
6. RECOMMENDATION
- Item embeddings
- User embeddings
- Collaborative filtering
7. KNOWLEDGE GRAPHS
- Entity embeddings
- Relation embeddings
- Link prediction
8. FEATURE EXTRACTION
- Preprocessing for ML
- Transfer learning
- Multi-task learning
""")
# FAMILY TREE
print("\n=== Embedding Methods Evolution ===")
print("""
1990s-2000s: One-hot, TF-IDF
- No semantic meaning
- High dimensional
- Sparse
2003: Neural Language Models (Bengio)
- First learned embeddings
- Slow to train
- Foundation for later work
2013: Word2Vec (Google)
- Skip-gram, CBOW
- Fast training
- Semantic analogies
2014: GloVe (Stanford)
- Global co-occurrence
- Fast, effective
- Widely used
2016: FastText (Facebook)
- Subword information
- Handles OOV
- Morphologically aware
2018: ELMo (AllenAI)
- Contextual embeddings
- Deep biLSTM
- Task-specific
2018: BERT (Google)
- Transformer-based
- Contextual
- SOTA
2019+: GPT, RoBERTa, T5, etc.
- Larger, better
- Multimodal
- Foundation models
""")๐ข One-Hot Encoding
๐ฏ The Simplest Representation
One-hot encoding represents each word as a sparse binary vector with 1 at the word's index and 0 elsewhere. It's simple but has major limitations.
How It Works
Vocabulary: List of all unique words. Dimension: Size of vocabulary. Encoding: 1 at word index, 0 elsewhere. Sparse: Mostly zeros. No similarity: All words equally different. Curse of dimensionality: Huge vectors.
๐ข Binary๐ Sparse๐ฏ Simpleโ ๏ธ Limited
# One-hot encoding
import numpy as np
from sklearn.preprocessing import OneHotEncoder
from tensorflow.keras.preprocessing.text import Tokenizer
from tensorflow.keras.utils import to_categorical
# 1. MANUAL ONE-HOT
def manual_one_hot(vocab, word):
"""Manually create one-hot vector."""
vocab_size = len(vocab)
word_to_idx = {w: i for i, w in enumerate(vocab)}
vector = np.zeros(vocab_size)
if word in word_to_idx:
vector[word_to_idx[word]] = 1
return vector
# Example
vocab = ['cat', 'dog', 'bird', 'fish', 'horse']
print("Vocabulary:", vocab)
print("Size:", len(vocab))
# One-hot for each word
print("\n=== One-Hot Vectors ===")
for word in vocab:
vector = manual_one_hot(vocab, word)
print(f"{word:8s}: {vector}")
# 2. WITH KERAS
texts = [
"I love machine learning",
"Deep learning is amazing",
"Natural language processing",
"Machine learning is powerful",
]
tokenizer = Tokenizer()
tokenizer.fit_on_texts(texts)
# Vocabulary
word_index = tokenizer.word_index
print(f"\nVocabulary size: {len(word_index)}")
print(f"Word index: {list(word_index.items())[:10]}")
# Convert to sequences
sequences = tokenizer.texts_to_sequences(texts)
print(f"\nSequences: {sequences}")
# One-hot encode
one_hot = tokenizer.texts_to_matrix(texts, mode='binary')
print(f"\nOne-hot shape: {one_hot.shape}")
print(f"Sparse: {np.count_nonzero(one_hot) / one_hot.size:.4f}")
# 3. PYTORCH ONE-HOT
import torch
import torch.nn.functional as F
def pytorch_one_hot(text, vocab):
"""PyTorch one-hot encoding."""
word_to_idx = {w: i for i, w in enumerate(vocab)}
indices = [word_to_idx.get(w, 0) for w in text.split()]
# One-hot via scatter
indices_tensor = torch.tensor(indices)
one_hot = F.one_hot(indices_tensor, num_classes=len(vocab))
return one_hot
# Example
text = "cat dog bird"
one_hot = pytorch_one_hot(text, vocab)
print(f"\nPyTorch one-hot shape: {one_hot.shape}")
print(one_hot)
# 4. LIMITATIONS
print("\n=== Limitations of One-Hot ===")
print("""
1. HIGH DIMENSIONALITY
- Vocabulary: 50,000 words
- Vector: 50,000 dimensions
- Memory: 200KB per word (float64)
- Total: 10GB for vocabulary!
2. NO SEMANTIC SIMILARITY
- "cat" and "kitten": orthogonal (dot = 0)
- "cat" and "car": orthogonal
- All words equally dissimilar
3. NO GENERALIZATION
- Model learns "cat" is positive
- Doesn't transfer to "kitten"
- Each word is independent
4. SPARSE REPRESENTATION
- 99.998% zeros
- Inefficient computation
- Wastes memory
5. CURSE OF DIMENSIONALITY
- Distance becomes meaningless
- Need exponential data
- Poor model performance
6. NO COMPOSITIONALITY
- Can't represent phrases
- "New York" vs "new" and "York"
- No word relationships
""")
# 5. WHEN TO USE
print("\n=== When to Use One-Hot ===")
print("""
Still useful when:
โ Very small vocabulary (< 100)
โ Categorical features (not text)
โ Simple bag-of-words models
โ Baseline comparison
โ One-hot labels for classification
Avoid when:
โ Large vocabulary
โ Semantic similarity needed
โ Deep learning models
โ Transfer learning
Legacy use cases:
- Naive Bayes
- Simple logistic regression
- Historical NLP systems
""")
# 6. COMPARISON
print("\n=== One-Hot vs Embeddings ===")
print(f"{'Aspect':<25} {'One-Hot':<20} {'Embeddings':<20}")
print("-" * 65)
print(f"{'Dimensionality':<25} {'Vocabulary size':<20} {'50-300':<20}")
print(f"{'Density':<25} {'Sparse':<20} {'Dense':<20}")
print(f"{'Similarity':<25} {'None':<20} {'Semantic':<20}")
print(f"{'Memory (50K vocab)':<25} {'~10 GB':<20} {'~60 MB':<20}")
print(f"{'Generalization':<25} {'No':<20} {'Yes':<20}")
print(f"{'Training':<25} {'N/A':<20} {'Learned':<20}")
print(f"{'Semantic analogies':<25} {'No':<20} {'Yes':<20}")
# 7. TRANSITION TO EMBEDDINGS
print("\n=== From One-Hot to Embeddings ===")
print("""
Key insight: Replace one-hot with learned dense vectors.
One-hot: [0, 0, 1, 0, 0, ..., 0] (50,000 dims)
โ Embedding matrix (50,000 ร 300)
Embedding: [0.2, -0.5, 0.8, ..., 0.1] (300 dims)
How it works:
1. Embedding matrix: V ร d (V = vocab, d = dim)
2. Lookup: one_hot @ embedding_matrix
3. Result: d-dimensional vector
Benefits:
- 300 dims vs 50,000
- Semantically meaningful
- Learned from data
- Can be pretrained
Implementation (PyTorch):
embedding = nn.Embedding(vocab_size, embed_dim)
vectors = embedding(word_indices) # (batch, seq_len, embed_dim)
""")๐งฌ Word2Vec (Skip-gram, CBOW)
๐ฏ The Breakthrough
Word2Vec (Mikolov et al., 2013) was a breakthrough in word embeddings. It learns vectors by predicting context words (Skip-gram) or target words (CBOW) from the surrounding text.
Two Architectures
Skip-gram: Predict context from target word. CBOW: Predict target from context. Window size: How many context words. Negative sampling: Efficient training. Hierarchical softmax: Alternative. Result: Semantic vectors.
๐ฏ Skip-gram๐ฎ CBOW๐ Windowโก Neg Sampling
# Word2Vec โ theory and implementation
import numpy as np
import torch
import torch.nn as nn
import torch.optim as optim
from collections import Counter
# 1. SKIP-GRAM ARCHITECTURE
class SkipGram(nn.Module):
"""
Skip-gram: predict context words from target word.
Example: "The cat sat on the mat"
Target: "cat"
Context: ["The", "sat"] (window=1)
"""
def __init__(self, vocab_size, embedding_dim):
super().__init__()
# Embedding layers
self.target_embedding = nn.Embedding(vocab_size, embedding_dim)
self.context_embedding = nn.Embedding(vocab_size, embedding_dim)
# Initialize
nn.init.xavier_uniform_(self.target_embedding.weight)
nn.init.xavier_uniform_(self.context_embedding.weight)
def forward(self, target, context):
"""
target: (batch,) โ target word indices
context: (batch,) โ context word indices
"""
target_emb = self.target_embedding(target) # (B, d)
context_emb = self.context_embedding(context) # (B, d)
# Dot product
score = (target_emb * context_emb).sum(dim=1) # (B,)
return score
# 2. CBOW ARCHITECTURE
class CBOW(nn.Module):
"""
CBOW: predict target word from context words.
Example: "The cat sat on the mat"
Context: ["The", "sat"]
Target: "cat"
"""
def __init__(self, vocab_size, embedding_dim):
super().__init__()
# Embedding layers
self.context_embedding = nn.Embedding(vocab_size, embedding_dim)
self.target_embedding = nn.Embedding(vocab_size, embedding_dim)
nn.init.xavier_uniform_(self.context_embedding.weight)
nn.init.xavier_uniform_(self.target_embedding.weight)
def forward(self, context, target):
"""
context: (batch, context_size) โ context word indices
target: (batch,) โ target word indices
"""
# Average context embeddings
context_emb = self.context_embedding(context) # (B, C, d)
context_avg = context_emb.mean(dim=1) # (B, d)
# Target embedding
target_emb = self.target_embedding(target) # (B, d)
# Dot product
score = (context_avg * target_emb).sum(dim=1) # (B,)
return score
# 3. NEGATIVE SAMPLING
def negative_sampling_loss(model, target, context, num_negatives, vocab_size, device):
"""
Negative sampling loss.
Instead of full softmax over vocabulary, only classify:
- Positive sample: (target, context)
- Negative samples: (target, random_words)
"""
batch_size = target.size(0)
# Positive scores
pos_score = model(target, context)
pos_loss = -torch.log(torch.sigmoid(pos_score) + 1e-8)
# Negative samples
neg_contexts = torch.randint(0, vocab_size, (batch_size, num_negatives), device=device)
# Expand target for negatives
target_expanded = target.unsqueeze(1).expand(-1, num_negatives)
# Negative scores
neg_score = model(target_expanded.reshape(-1), neg_contexts.reshape(-1))
neg_score = neg_score.reshape(batch_size, num_negatives)
neg_loss = -torch.log(torch.sigmoid(-neg_score) + 1e-8).sum(dim=1)
# Total loss
loss = (pos_loss + neg_loss).mean()
return loss
# 4. FULL TRAINING
class Word2VecTrainer:
def __init__(self, vocab_size, embedding_dim=100,
window_size=2, num_negatives=5, lr=0.025):
self.vocab_size = vocab_size
self.embedding_dim = embedding_dim
self.window_size = window_size
self.num_negatives = num_negatives
self.model = SkipGram(vocab_size, embedding_dim)
self.optimizer = optim.Adam(self.model.parameters(), lr=lr)
def train(self, corpus, epochs=5, batch_size=512):
"""Train Word2Vec on corpus."""
# Build vocabulary
word_counts = Counter(corpus)
vocab = {w: i for i, (w, _) in enumerate(word_counts.most_common())}
idx_to_word = {i: w for w, i in vocab.items()}
# Generate training pairs
pairs = []
tokens = corpus.split()
for i, word in enumerate(tokens):
if word not in vocab:
continue
# Context window
for j in range(max(0, i - self.window_size),
min(len(tokens), i + self.window_size + 1)):
if i != j and tokens[j] in vocab:
pairs.append((vocab[word], vocab[tokens[j]]))
# Train
for epoch in range(epochs):
np.random.shuffle(pairs)
total_loss = 0
for i in range(0, len(pairs), batch_size):
batch = pairs[i:i + batch_size]
targets = torch.tensor([p[0] for p in batch])
contexts = torch.tensor([p[1] for p in batch])
self.optimizer.zero_grad()
loss = negative_sampling_loss(
self.model, targets, contexts,
self.num_negatives, self.vocab_size, 'cpu'
)
loss.backward()
self.optimizer.step()
total_loss += loss.item()
print(f"Epoch {epoch+1}: Loss = {total_loss / len(pairs):.4f}")
return self.model, vocab, idx_to_word
def get_embeddings(self):
"""Get learned embeddings."""
return self.model.target_embedding.weight.detach().numpy()
# 5. USING PRETRAINED WORD2VEC
print("=== Using Pretrained Word2Vec ===")
print("""
# Google News (300-dim, 3M words)
from gensim.models import KeyedVectors
w2v = KeyedVectors.load_word2vec_format(
'GoogleNews-vectors-negative300.bin', binary=True
)
# Similar words
similar = w2v.most_similar('king', topn=5)
for word, score in similar:
print(f" {word}: {score:.4f}")
# Analogy: king - man + woman = ?
result = w2v.most_similar(
positive=['king', 'woman'],
negative=['man'],
topn=3
)
# Similarity
similarity = w2v.similarity('cat', 'kitten')
print(f"cat-kitten similarity: {similarity:.4f}")
""")
# 6. KEY PARAMETERS
print("\n=== Word2Vec Parameters ===")
print("""
VECTOR SIZE (dimensions):
- 50-100: Fast, less accurate
- 100-300: Standard
- 300-500: Rich representations
- Higher: Diminishing returns
WINDOW SIZE:
- 2-5: Syntactic similarity
- 5-10: Semantic similarity
- Larger: Broader context
- Rule of thumb: 5-10 for semantic
MIN COUNT:
- Minimum word frequency
- 1-5 typical
- Higher: Cleaner but misses rare words
NEGATIVE SAMPLING:
- Number of negatives
- 5-20 typical
- More: Slower but better
EPOCHS:
- 3-10 typical
- More: Better but risk overfitting
LEARNING RATE:
- 0.025 initial (Mikolov)
- Linear decay to 0.0001
""")
# 7. SKIP-GRAM VS CBOW
print("\n=== Skip-gram vs CBOW ===")
print(f"{'Aspect':<25} {'Skip-gram':<20} {'CBOW':<20}")
print("-" * 65)
print(f"{'Task':<25} {'Predict context':<20} {'Predict target':<20}")
print(f"{'Input':<25} {'Target word':<20} {'Context words':<20}")
print(f"{'Output':<25} {'Context words':<20} {'Target word':<20}")
print(f"{'Training speed':<25} {'Slower':<20} {'Faster':<20}")
print(f"{'Small data':<25} {'Better':<20} {'Worse':<20}")
print(f"{'Large data':<25} {'Slightly better':<20} {'Faster':<20}")
print(f"{'Rare words':<25} {'Better':<20} {'Worse':<20}")
print(f"{'Frequent words':<25} {'Worse':<20} {'Better':<20}")
# 8. WHY IT WORKS
print("\n=== Why Word2Vec Works ===")
print("""
Distributional Hypothesis:
"You shall know a word by the company it keeps"
โ J.R. Firth (1957)
Key insight:
Words in similar contexts have similar meanings.
Example:
"The cat sat on the mat"
"The dog sat on the rug"
'cat' and 'dog' both appear in "sat on the" context
โ They get similar embeddings
Implicit factorization:
Word2Vec implicitly factorizes the PMI matrix
(Levy & Goldberg, 2014)
Where:
PMI(w, c) = log(P(w,c) / (P(w) * P(c)))
This explains why it captures semantics so well.
""")
# 9. ADVANTAGES AND LIMITATIONS
print("\n=== Word2Vec: Pros and Cons ===")
print("""
ADVANTAGES:
โ Fast training (hours on large corpus)
โ Semantic relationships
โ Works well in practice
โ Pretrained models available
โ Simple and interpretable
โ Handles analogies
LIMITATIONS:
โ One vector per word (no context)
โ Cannot handle polysemy
โ Ignores word order (partially)
โ No subword information (OOV)
โ Static embeddings
โ Bias in training data
Modern alternatives:
- GloVe (better optimization)
- FastText (subword info)
- ELMo, BERT (contextual)
- Universal Sentence Encoder
- Sentence-BERT
""")๐ GloVe
๐ฏ Global Vectors for Word Representation
GloVe (Pennington et al., 2014) is a word embedding method based on global word-word co-occurrence statistics. Unlike Word2Vec's local context windows, GloVe uses the entire corpus's co-occurrence matrix.
How GloVe Works
1. Co-occurrence: Count word pairs. 2. Matrix: V ร V co-occurrence matrix. 3. Objective: Factorize log co-occurrences. 4. Weighted: Weight by frequency. 5. Training: Gradient descent. 6. Result: Word vectors.
๐ Global๐ Co-occurrence๐ Factorizationโ๏ธ Weighted
# GloVe โ theory and implementation
import numpy as np
import torch
import torch.nn as nn
import torch.optim as optim
from collections import defaultdict
# 1. GLOVE OBJECTIVE
print("=== GloVe Objective ===")
print("""
GloVe models word-word co-occurrences:
w_i ยท w_j + b_i + b_j = log(X_ij)
Where:
w_i, w_j: word vectors
b_i, b_j: bias terms
X_ij: number of times word j appears in context of word i
Loss function:
J = ฮฃ_{i,j} f(X_ij) (w_i ยท w_j + b_i + b_j - log X_ij)ยฒ
Weighting function:
f(x) = (x / x_max)^0.75 if x < x_max
= 1 if x โฅ x_max
Key insight:
Ratio of co-occurrence probabilities encodes meaning.
P(k|ice) / P(k|steam) is large for k='solid'
P(k|ice) / P(k|steam) is small for k='gas'
P(k|ice) / P(k|steam) โ 1 for k='water' or 'fashion'
""")
# 2. GLOVE IMPLEMENTATION
class GloVe(nn.Module):
"""GloVe model."""
def __init__(self, vocab_size, embedding_dim, x_max=100, alpha=0.75):
super().__init__()
self.vocab_size = vocab_size
self.embedding_dim = embedding_dim
self.x_max = x_max
self.alpha = alpha
# Word vectors
self.word_vectors = nn.Embedding(vocab_size, embedding_dim)
self.context_vectors = nn.Embedding(vocab_size, embedding_dim)
# Biases
self.word_bias = nn.Embedding(vocab_size, 1)
self.context_bias = nn.Embedding(vocab_size, 1)
# Initialize
nn.init.uniform_(self.word_vectors.weight, -0.5, 0.5)
nn.init.uniform_(self.context_vectors.weight, -0.5, 0.5)
nn.init.zeros_(self.word_bias.weight)
nn.init.zeros_(self.context_bias.weight)
def forward(self, word_idx, context_idx):
"""Predict log co-occurrence."""
w = self.word_vectors(word_idx) # (B, d)
c = self.context_vectors(context_idx) # (B, d)
b_w = self.word_bias(word_idx).squeeze(-1) # (B,)
b_c = self.context_bias(context_idx).squeeze(-1) # (B,)
# Prediction
prediction = (w * c).sum(dim=1) + b_w + b_c
return prediction
def loss(self, word_idx, context_idx, cooccurrence):
"""GloVe loss."""
prediction = self.forward(word_idx, context_idx)
target = torch.log(cooccurrence + 1e-8)
# Weighting function
weight = torch.clamp((cooccurrence / self.x_max) ** self.alpha, max=1.0)
# Weighted squared loss
loss = weight * (prediction - target) ** 2
return loss.mean()
# 3. BUILD CO-OCCURRENCE MATRIX
def build_cooccurrence(corpus, vocab, window_size=5):
"""Build co-occurrence matrix."""
vocab_size = len(vocab)
cooccurrence = defaultdict(float)
tokens = corpus.split()
for i, word in enumerate(tokens):
if word not in vocab:
continue
for j in range(max(0, i - window_size),
min(len(tokens), i + window_size + 1)):
if i != j and tokens[j] in vocab:
# Distance weighting
distance = abs(i - j)
weight = 1.0 / distance
w_idx = vocab[word]
c_idx = vocab[tokens[j]]
cooccurrence[(w_idx, c_idx)] += weight
return cooccurrence
# 4. TRAINING GLOVE
def train_glove(corpus, embedding_dim=100, window_size=5,
epochs=25, batch_size=512, lr=0.05):
"""Train GloVe embeddings."""
from collections import Counter
# Build vocabulary
words = corpus.split()
word_counts = Counter(words)
vocab = {w: i for i, (w, _) in enumerate(word_counts.most_common())}
print(f"Vocabulary size: {len(vocab)}")
# Build co-occurrence
cooccurrence = build_cooccurrence(corpus, vocab, window_size)
print(f"Co-occurrence pairs: {len(cooccurrence)}")
# Convert to tensors
pairs = list(cooccurrence.keys())
counts = torch.tensor([cooccurrence[p] for p in pairs])
word_indices = torch.tensor([p[0] for p in pairs])
context_indices = torch.tensor([p[1] for p in pairs])
# Model
model = GloVe(len(vocab), embedding_dim)
optimizer = optim.Adam(model.parameters(), lr=lr)
# Training
n_pairs = len(pairs)
for epoch in range(epochs):
# Shuffle
perm = torch.randperm(n_pairs)
total_loss = 0
for i in range(0, n_pairs, batch_size):
idx = perm[i:i + batch_size]
optimizer.zero_grad()
loss = model.loss(
word_indices[idx],
context_indices[idx],
counts[idx]
)
loss.backward()
optimizer.step()
total_loss += loss.item()
avg_loss = total_loss / (n_pairs / batch_size)
print(f"Epoch {epoch+1}: Loss = {avg_loss:.4f}")
return model, vocab
# 5. USING PRETRAINED GLOVE
print("\n=== Using Pretrained GloVe ===")
print("""
# Download from Stanford: https://nlp.stanford.edu/projects/glove/
# Files: glove.6B.zip, glove.42B.zip, glove.840B.zip
from gensim.models import KeyedVectors
from gensim.scripts.glove2word2vec import glove2word2vec
# Convert GloVe to Word2Vec format
glove_input = 'glove.6B.100d.txt'
word2vec_output = 'glove.6B.100d.word2vec'
glove2word2vec(glove_input, word2vec_output)
# Load
glove = KeyedVectors.load_word2vec_format(word2vec_output, binary=False)
# Use
print(f"Vocabulary: {len(glove)}")
print(f"Dimensions: {glove.vector_size}")
# Similar words
similar = glove.most_similar('king', topn=5)
for word, score in similar:
print(f" {word}: {score:.4f}")
# Similarity
print(f"cat-dog: {glove.similarity('cat', 'dog'):.4f}")
print(f"cat-computer: {glove.similarity('cat', 'computer'):.4f}")
""")
# 6. WORD2VEC VS GLOVE
print("\n=== Word2Vec vs GloVe ===")
print(f"{'Aspect':<25} {'Word2Vec':<20} {'GloVe':<20}")
print("-" * 65)
print(f"{'Approach':<25} {'Predictive':<20} {'Count-based':<20}")
print(f"{'Training':<25} {'Local windows':<20} {'Global matrix':<20}")
print(f"{'Objective':<25} {'Negative sampling':<20} {'Weighted LS':<20}")
print(f"{'Speed':<25} {'Fast':<20} {'Fast':<20}")
print(f"{'Performance':<25} {'Good':<20} {'Good':<20}")
print(f"{'Theory':<25} {'Implicit PMI':<20} {'Explicit PMI':<20}")
print(f"{'Implementation':<25} {'SGNS':<20} {'Matrix factor':<20}")
print(f"{'Pretrained':<25} {'GoogleNews':<20} {'Wikipedia':<20}")
# 7. GLOVE DATASETS
print("\n=== GloVe Pretrained Models ===")
print("""
glove.6B (Wikipedia 2014 + Gigaword 5):
- 400K vocabulary
- 6B tokens
- Dimensions: 50, 100, 200, 300
- Size: ~800 MB total
glove.42B (Common Crawl):
- 1.9M vocabulary
- 42B tokens
- Dimensions: 300
- Size: ~5 GB
glove.840B (Common Crawl):
- 2.2M vocabulary
- 840B tokens
- Dimensions: 300
- Size: ~7 GB
glove.twitter.27B:
- 1.2M vocabulary
- 27B tokens from Twitter
- Dimensions: 25, 50, 100, 200
- Good for social media
glove.6B.300d most common:
- Best general-purpose
- Good for most tasks
""")
# 8. ADVANTAGES AND LIMITATIONS
print("\n=== GloVe: Pros and Cons ===")
print("""
ADVANTAGES:
โ Global corpus statistics
โ Faster than Word2Vec in some cases
โ Good performance
โ Multiple pretrained sizes
โ Well-understood theory
โ Efficient memory
LIMITATIONS:
โ Static embeddings (like Word2Vec)
โ No polysemy handling
โ No subword info (OOV issue)
โ Requires co-occurrence matrix
โ Memory for large vocab
โ Bias in corpus
Compare to Word2Vec:
- Both capture similar relationships
- GloVe slightly better on analogy tasks
- Word2Vec slightly better on similarity
- Choice depends on task
Modern alternatives:
- ELMo, BERT (contextual)
- FastText (subword)
- Sentence-BERT (sentences)
""")
# 9. PRACTICAL TIPS
print("\n=== Practical Tips ===")
print("""
1. USE PRETRAINED GLOVE
- glove.6B.100d or glove.6B.300d
- Covers most English words
- Saves training time
2. FINE-TUNE FOR DOMAIN
- Start with pretrained
- Continue training on domain data
- Improves task performance
3. CHOOSE DIMENSIONS
- 50: Mobile, fast
- 100: Standard
- 300: Best quality
- Depends on resources
4. HANDLE OOV
- Map to token
- Random initialization
- Or use FastText
5. COMBINE WITH CONTEXT
- GloVe + BiLSTM
- GloVe + Transformer
- Better than static alone
6. VISUALIZE
- t-SNE or UMAP
- Check semantic clusters
- Debug training
7. EVALUATE
- Word similarity tasks
- Analogy tasks
- Task-specific metrics
""") โก FastText
๐ฏ Embeddings with Subword Information
FastText (Bojanowski et al., 2016) improves on Word2Vec by representing each word as a bag of character n-grams. This handles rare words, misspellings, and morphological variations.
Subword Information
Idea: Words composed of character n-grams. Example: "running" โ run, unn, nni, nin, ing + full word. Benefit: Handles OOV, misspellings. Morphology: Captures prefixes/suffixes. Shared: Similar words share n-grams. Better: For morphologically rich languages.
๐ค Subword๐งฉ N-grams๐ OOV๐ Morphology
# FastText โ subword embeddings
import numpy as np
import torch
import torch.nn as nn
from collections import Counter
# 1. CHARACTER N-GRAMS
print("=== Character N-grams ===")
print("""
Word: "running"
Character n-grams (n=3):
Full word:
Boundary:
More:
Common n-grams with "runs":
- "run" (3-gram)
- "uns" (3-gram)
Common n-grams with "runner":
- "run"
- "unn"
- "nne"
- "ner"
Shared subwords โ shared meaning!
""")
# 2. FASTTEXT ARCHITECTURE
class FastText(nn.Module):
"""
FastText model with subword embeddings.
Word vector = average of its n-gram vectors + word vector
"""
def __init__(self, vocab_size, embedding_dim, num_buckets=2_000_000,
min_n=3, max_n=6):
super().__init__()
self.embedding_dim = embedding_dim
self.min_n = min_n
self.max_n = max_n
self.num_buckets = num_buckets
# Word embeddings
self.word_embedding = nn.Embedding(vocab_size, embedding_dim)
# Subword (n-gram) embeddings (hashed)
self.ngram_embedding = nn.Embedding(num_buckets, embedding_dim)
# Initialize
nn.init.uniform_(self.word_embedding.weight, -0.05, 0.05)
nn.init.uniform_(self.ngram_embedding.weight, -0.05, 0.05)
def get_ngrams(self, word):
"""Extract character n-grams from word."""
word = f"<{word}>"
ngrams = []
for n in range(self.min_n, self.max_n + 1):
for i in range(len(word) - n + 1):
ngrams.append(word[i:i+n])
return ngrams
def hash_ngram(self, ngram):
"""Hash n-gram to bucket index."""
return hash(ngram) % self.num_buckets
def get_word_vector(self, word_idx, word):
"""Compute word vector = word + average of n-grams."""
# Word vector
word_vec = self.word_embedding(word_idx)
# N-gram vectors
ngrams = self.get_ngrams(word)
ngram_indices = torch.tensor([self.hash_ngram(ng) for ng in ngrams])
ngram_vecs = self.ngram_embedding(ngram_indices)
# Average n-grams
avg_ngram_vec = ngram_vecs.mean(dim=0)
# Combine
return word_vec + avg_ngram_vec
def forward(self, target_idx, context_idx, target_word, context_word):
"""Compute similarity score."""
target_vec = self.get_word_vector(target_idx, target_word)
context_vec = self.get_word_vector(context_idx, context_word)
return (target_vec * context_vec).sum()
# 3. WHY SUBWORDS HELP
print("\n=== Why Subwords Help ===")
print("""
Problem: Traditional embeddings fail on:
1. OOV (Out-of-Vocabulary)
- New word: "selfie" not in training
- Standard: vector (no info)
- FastText: composed of known n-grams
2. Misspellings
- "restarant" instead of "restaurant"
- Standard: wrong vector
- FastText: similar n-grams โ similar vector
3. Morphological variations
- "run", "runs", "running", "runner"
- Standard: separate words
- FastText: share "run" n-gram
4. Compound words
- "sunflower" vs "sun" and "flower"
- Standard: no relationship
- FastText: shares n-grams
5. Rare words
- "syzygy" (rare)
- Standard: poor vector
- FastText: n-grams help
""")
# 4. USING PRETRAINED FASTTEXT
print("\n=== Using Pretrained FastText ===")
print("""
# Install: pip install fasttext
import fasttext
import fasttext.util
# Download pretrained English (or other languages)
fasttext.util.download_model('en', if_exists='ignore') # 1M words
ft = fasttext.load_model('cc.en.300.bin')
# Get word vector
vector = ft.get_word_vector('hello')
print(f"Vector shape: {vector.shape}")
# Similar words
similar = ft.get_nearest_neighbors('hello', k=5)
print("Similar words:")
for score, word in similar:
print(f" {word}: {score:.4f}")
# Out-of-vocabulary (OOV) - works!
oov_vector = ft.get_word_vector('antidisestablishmentarianism')
print(f"OOV vector computed: {oov_vector.shape}")
# Analogy
def analogy(w1, w2, w3, model, k=3):
vec = model.get_word_vector(w1) - model.get_word_vector(w2) + model.get_word_vector(w3)
# Find closest
# ...
# Language identification
model = fasttext.load_model('lid.176.bin')
predictions = model.predict("Bonjour le monde", k=3)
print(f"Language: {predictions}")
""")
# 5. FASTTEXT FOR CLASSIFICATION
print("\n=== FastText for Classification ===")
print("""
# FastText is also a text classifier!
# Prepare data (fastText format)
# __label__positive This is a great product!
# __label__negative This is terrible.
import fasttext
# Train classifier
model = fasttext.train_supervised(
input='train.txt',
epoch=25,
lr=1.0,
wordNgrams=2,
dim=100,
minCount=1,
loss='softmax'
)
# Predict
predictions = model.predict("This movie is amazing!")
print(f"Label: {predictions[0]}")
print(f"Probability: {predictions[1]}")
# Evaluate
result = model.test('test.txt')
print(f"Precision: {result[1]}")
print(f"Recall: {result[2]}")
""")
# 6. COMPARISON
print("\n=== Word2Vec vs GloVe vs FastText ===")
print(f"{'Aspect':<25} {'Word2Vec':<15} {'GloVe':<15} {'FastText':<15}")
print("-" * 70)
print(f"{'Subwords':<25} {'No':<15} {'No':<15} {'Yes':<15}")
print(f"{'OOV handling':<25} {'':<15} {'':<15} {'Works':<15}")
print(f"{'Misspellings':<25} {'Poor':<15} {'Poor':<15} {'Good':<15}")
print(f"{'Morphology':<25} {'No':<15} {'No':<15} {'Yes':<15}")
print(f"{'File size (EN)':<25} {'~1.5GB':<15} {'~800MB':<15} {'~7GB':<15}")
print(f"{'Training speed':<25} {'Fast':<15} {'Fast':<15} {'Slower':<15}")
print(f"{'Memory':<25} {'Low':<15} {'Low':<15} {'High':<15}")
# 7. PRACTICAL USES
print("\n=== FastText Practical Uses ===")
print("""
1. MULTILINGUAL NLP
- 157 languages available
- Cross-lingual transfer
- Language-specific patterns
2. MORPHOLOGICALLY RICH LANGUAGES
- German compound words
- Turkish agglutination
- Finnish cases
- Arabic roots
3. SOCIAL MEDIA
- Handles misspellings
- Slang variations
- Creative spellings
4. DOMAIN-SPECIFIC
- Medical terminology
- Chemical names
- Biological terms
5. LOW-RESOURCE LANGUAGES
- Can train on small corpora
- Subwords help generalize
- Better than Word2Vec
6. TEXT CLASSIFICATION
- FastText classifier
- Real-time inference
- Competitive accuracy
7. LANGUAGE IDENTIFICATION
- 176 languages
- Very fast
- High accuracy
""")
# 8. LIMITATIONS
print("\n=== FastText Limitations ===")
print("""
1. LARGER MODEL SIZE
- N-gram embeddings add size
- 7GB for full English
- Slower to load
2. SLOWER TRAINING
- More parameters
- More computation
- But still reasonable
3. MEMORY USAGE
- More RAM needed
- Higher hardware requirements
4. STILL STATIC
- No context awareness
- One vector per word (plus ngrams)
- Same limitation as Word2Vec
5. HASHING COLLISIONS
- Different n-grams โ same bucket
- Can introduce noise
- Usually acceptable
Modern alternative:
- BERT and contextual models
- Subword tokenization (BPE, WordPiece)
- Different approach, same benefits
""") ๐ Semantic Analogies
๐ฏ Vector Arithmetic for Meaning
One of the most remarkable properties of word embeddings is that they support semantic analogies via vector arithmetic. king - man + woman โ queen.
The Magic of Vector Arithmetic
Classic: king - man + woman = queen. Geography: Paris - France + Italy = Rome. Tense: walking - walk + swim = swimming. Company: Apple - iPhone + Samsung = Galaxy. Why: Consistent directions encode meaning. Property: Linear structure in vector space.
๐ Analogy๐บ๏ธ Geographyโฐ Tenseโ Arithmetic
# Semantic analogies with word embeddings
import numpy as np
from sklearn.metrics.pairwise import cosine_similarity
# 1. CLASSIC ANALOGY METHOD
def analogy(word1, word2, word3, embeddings, word_to_idx, idx_to_word, topn=5):
"""
Solve analogy: word1 - word2 + word3 = ?
Example: king - man + woman = queen
"""
# Get vectors
v1 = embeddings[word_to_idx[word1]]
v2 = embeddings[word_to_idx[word2]]
v3 = embeddings[word_to_idx[word3]]
# Compute target vector
target = v1 - v2 + v3
# Compute similarity to all words
similarities = cosine_similarity([target], embeddings)[0]
# Exclude input words
for w in [word1, word2, word3]:
if w in word_to_idx:
similarities[word_to_idx[w]] = -1
# Get top-N most similar
top_indices = np.argsort(similarities)[-topn:][::-1]
results = [(idx_to_word[i], similarities[i]) for i in top_indices]
return results
# 2. EXAMPLES OF ANALOGIES
print("=== Classic Analogies ===")
print("""
Semantic relationships captured:
1. GENDER
king - man + woman = queen
actor - man + woman = actress
uncle - man + woman = aunt
brother - man + woman = sister
2. PLURALITY
cat - cats + dog = dogs
car - cars + bus = buses
3. TENSE
walking - walk + swim = swimming
went - go + eat = ate
4. COUNTRY-CAPITAL
Paris - France + Italy = Rome
Berlin - Germany + Spain = Madrid
Tokyo - Japan + China = Beijing
5. CURRENCY
dollar - USA + Japan = yen
euro - France + UK = pound
6. COMPANY-PRODUCT
iPhone - Apple + Samsung = Galaxy
Windows - Microsoft + Apple = macOS
7. ADJECTIVE-ADVERB
quick - quickly + slow = slowly
happy - happily + sad = sadly
8. COMPARATIVE-SUPERLATIVE
good - better + bad = worse
big - bigger + small = smaller
""")
# 3. WHY IT WORKS
print("\n=== Why Analogies Work ===")
print("""
Mathematical explanation:
Word embeddings capture linear substructures!
Example: Gender direction
queen - king โ woman - man โ female direction
If:
king = [0.9, 0.1, 0.8, ...]
man = [0.8, 0.2, 0.7, ...]
woman = [0.8, 0.3, 0.9, ...]
Then:
king - man = [0.1, -0.1, 0.1, ...] (male direction)
queen - woman โ same direction
Therefore:
king - man + woman = queen
This means:
- Relationships are encoded as directions
- Consistent across word pairs
- Emergent property of training
Not designed, but emerges!
""")
# 4. IMPLEMENTATION WITH GENSIM
print("\n=== Implementation with Gensim ===")
print("""
from gensim.models import KeyedVectors
# Load pretrained
model = KeyedVectors.load_word2vec_format(
'GoogleNews-vectors-negative300.bin', binary=True
)
# Analogy
result = model.most_similar(
positive=['king', 'woman'],
negative=['man'],
topn=3
)
print("king - man + woman =")
for word, score in result:
print(f" {word}: {score:.4f}")
# Multiple analogies
analogies = [
(['king', 'woman'], ['man'], 'queen'),
(['Paris', 'Italy'], ['France'], 'Rome'),
(['walking', 'swim'], ['walk'], 'swimming'),
(['bigger', 'small'], ['big'], 'smaller'),
]
for positive, negative, expected in analogies:
result = model.most_similar(positive=positive, negative=negative, topn=1)
predicted = result[0][0]
correct = "โ" if predicted == expected else "โ"
print(f"{correct} {positive} - {negative} = {predicted} (expected {expected})")
""")
# 5. USING EMBEDDINGS FOR ANALOGY
from sklearn.metrics.pairwise import cosine_similarity
def evaluate_analogies(embeddings, word_to_idx, idx_to_word,
analogy_pairs, topn=1):
"""
Evaluate analogies.
analogy_pairs: list of (word1, word2, word3, expected)
"""
correct = 0
total = 0
for w1, w2, w3, expected in analogy_pairs:
if not all(w in word_to_idx for w in [w1, w2, w3, expected]):
continue
# Compute target
target = (embeddings[word_to_idx[w1]] -
embeddings[word_to_idx[w2]] +
embeddings[word_to_idx[w3]])
# Similarity
sims = cosine_similarity([target], embeddings)[0]
# Exclude inputs
for w in [w1, w2, w3]:
if w in word_to_idx:
sims[word_to_idx[w]] = -1
# Top prediction
top_idx = np.argmax(sims)
predicted = idx_to_word[top_idx]
if predicted == expected:
correct += 1
total += 1
return correct / total if total > 0 else 0
# Example analogy pairs
analogies = [
('king', 'man', 'woman', 'queen'),
('actor', 'man', 'woman', 'actress'),
('Paris', 'France', 'Italy', 'Rome'),
('Berlin', 'Germany', 'Spain', 'Madrid'),
('walking', 'walk', 'swim', 'swimming'),
('bigger', 'big', 'small', 'smaller'),
]
# Simulate evaluation
print("\n=== Analogy Evaluation ===")
print("""
Standard benchmark: Google analogy set
- 19,544 questions
- 14 relationship types
- Covers semantic and syntactic
- Word2Vec: ~70% accuracy
- GloVe: ~75% accuracy
- FastText: ~78% accuracy
- BERT: varies (contextual)
""")
# 6. LIMITATIONS
print("\n=== Analogy Limitations ===")
print("""
1. CULTURAL BIAS
- Reflects training data biases
- E.g., "doctor - man + woman" might give "nurse"
- Sensitive to be careful
2. CONTEXT IGNORANCE
- "bank" ambiguous (river/money)
- Static embeddings don't handle
- Same vector for all uses
3. FAILURES ON SOME ANALOGIES
- Not all relationships are linear
- Complex relationships fail
- Depends on training data
4. OOV WORDS
- Can't handle unseen words
- FastText helps but not perfect
5. LANGUAGE SPECIFIC
- Analogies vary by language
- English has many benchmarks
- Other languages less tested
6. EVALUATION ISSUES
- Multiple correct answers possible
- "queen" vs "monarch"
- Precision metric misses this
""")
# 7. MODERN ANALOGIES
print("\n=== Modern Analogy Benchmarks ===")
print("""
Google Analogy Test:
- 19,544 pairs
- 14 relationship types
- Standard benchmark
BATS (Bigger Analogy Test Set):
- 98,000 pairs
- 40 categories
- Better coverage
SemEval-2012 Task 2:
- Relational similarity
- Human judgments
WordSim-353:
- 353 word pairs
- Human similarity ratings
SimLex-999:
- 999 word pairs
- Focuses on similarity (not relatedness)
- Harder benchmark
Modern contextual models:
- BERT analogies via embeddings
- Variable performance
- Not directly comparable
""")
# 8. PRACTICAL USES
print("\n=== Practical Uses of Analogies ===")
print("""
1. MODEL EVALUATION
- Quick sanity check
- Compare embedding quality
- Standard benchmark
2. SEMANTIC SEARCH
- Query expansion
- Synonym finding
- Related concepts
3. RECOMMENDATION
- Item similarity
- User similarity
- Cross-domain transfer
4. KNOWLEDGE DISCOVERY
- Find implicit relationships
- Discover patterns
- Generate hypotheses
5. CREATIVE APPLICATIONS
- Poetry generation
- Story generation
- Word games
6. EDUCATION
- Vocabulary learning
- Concept mapping
- Language teaching
7. QUERY UNDERSTANDING
- Expand search queries
- Find related terms
- Improve recall
""")
# 9. VISUALIZATION
print("\n=== Visualizing Analogy Structure ===")
print("""
When visualized with PCA or t-SNE:
Vectors:
king โโโโโโ queen
man โโโโโโ woman
uncle โโโโโโ aunt
Parallel lines appear!
Same direction = same relationship
This shows:
- Linear structure in embedding space
- Meaning as directions
- Emergent from training
Applications:
- Understanding embeddings
- Debugging training
- Explaining to stakeholders
""")๐ญ Contextual Embeddings (ELMo)
๐ฏ Context Matters
Contextual embeddings generate different vectors for the same word depending on its context. ELMo (Peters et al., 2018) was the first widely adopted contextual embedding model.
Context-Dependent Vectors
Static: One vector per word. Contextual: Different vector per context. Example: "bank" in river vs finance. Model: Deep biLSTM. Character-based: Handles OOV. Layers: Different layers โ different info.
๐ญ Contextual๐ Bidirectional๐ Layer-wise๐ OOV
# ELMo and contextual embeddings
import numpy as np
import torch
import torch.nn as nn
# 1. WHY CONTEXT MATTERS
print("=== Why Context Matters ===")
print("""
Problem with static embeddings:
Word: "bank"
"I sat by the bank of the river." โ river bank
"I went to the bank to deposit money." โ financial bank
"The bank approved my loan." โ financial institution
Static embeddings:
Same vector for all uses of "bank"
Cannot distinguish meanings
Poor performance on contextual tasks
Contextual embeddings:
Different vector based on context
Captures meaning in context
Much better performance
Other polysemous words:
- "bat" (animal / sports equipment)
- "light" (illumination / not heavy)
- "spring" (season / water / coil)
- "right" (direction / correct / entitlement)
""")
# 2. ELMO ARCHITECTURE
class ELMo(nn.Module):
"""
ELMo: Embeddings from Language Models.
Architecture:
1. Character CNN for input (handles OOV)
2. Two-layer biLSTM
3. Weighted sum of layers for task
Trained on:
- Forward LM: predict next word
- Backward LM: predict previous word
"""
def __init__(self, vocab_size, char_vocab_size,
char_embed_dim=16, hidden_dim=512, num_layers=2):
super().__init__()
# Character embeddings
self.char_embedding = nn.Embedding(char_vocab_size, char_embed_dim)
# Character CNN
self.char_cnn = nn.Sequential(
nn.Conv1d(char_embed_dim, 32, kernel_size=5, padding=2),
nn.ReLU(),
nn.Conv1d(32, 32, kernel_size=3, padding=1),
nn.ReLU(),
nn.Conv1d(32, 64, kernel_size=3, padding=1),
nn.ReLU(),
)
# Project to hidden dim
self.char_proj = nn.Linear(64, hidden_dim)
# BiLSTM layers
self.lstm_layers = nn.ModuleList([
nn.LSTM(hidden_dim if i == 0 else hidden_dim * 2,
hidden_dim, batch_first=True, bidirectional=True)
for i in range(num_layers)
])
def forward(self, char_ids):
"""
char_ids: (B, seq_len, char_len) โ character IDs for each word
"""
B, seq_len, char_len = char_ids.shape
# Character embeddings
char_emb = self.char_embedding(char_ids) # (B, seq_len, char_len, char_emb_dim)
char_emb = char_emb.view(B * seq_len, char_len, -1).transpose(1, 2)
# Character CNN
char_features = self.char_cnn(char_emb) # (B*seq_len, 64, char_len)
char_features = char_features.max(dim=2)[0] # (B*seq_len, 64)
# Project
word_emb = self.char_proj(char_features).view(B, seq_len, -1)
# BiLSTM
outputs = [word_emb]
x = word_emb
for lstm in self.lstm_layers:
x, _ = lstm(x)
outputs.append(x)
# Return all layer outputs
return outputs # List of (B, seq_len, hidden_dim)
# 3. USING PRETRAINED ELMO
print("\n=== Using Pretrained ELMo ===")
print("""
# Install: pip install allennlp
from allennlp.commands.elmo import ElmoEmbedder
# Load pretrained ELMo
elmo = ElmoEmbedder(
options_file='https://allennlp.s3.amazonaws.com/models/elmo/2x4096_512_2048cnn_2xhighway/elmo_2x4096_512_2048cnn_2xhighway_options.json',
weight_file='https://allennlp.s3.amazonaws.com/models/elmo/2x4096_512_2048cnn_2xhighway/elmo_2x4096_512_2048cnn_2xhighway_weights.hdf5'
)
# Get embeddings for sentence
sentence = ["The", "bank", "by", "the", "river"]
embeddings = elmo.embed_sentence(sentence)
print(f"Shape: {embeddings.shape}")
# (3 layers, num_words, 1024)
# Layer 0: character embeddings (context-free)
# Layer 1: first biLSTM (syntactic)
# Layer 2: second biLSTM (semantic)
# Task-specific weighted combination
# Different tasks benefit from different layers
""")
# 4. LAYER-WISE INFORMATION
print("\n=== ELMo Layer Information ===")
print("""
Different layers capture different information:
Layer 0 (Character CNN):
- Character-level features
- Handles OOV
- Basic subword info
Layer 1 (First BiLSTM):
- Syntactic information
- POS tags, syntax
- Local context
Layer 2 (Second BiLSTM):
- Semantic information
- Word sense
- Long-range context
Task-specific weights:
- NER: prefers layer 2 (semantic)
- POS tagging: prefers layer 1 (syntax)
- Coreference: middle layers
- Sentiment: layer 2
Learnable weights:
ELMo = ฮณ * ฮฃ_t s_t * h_t
where s_t are task-specific softmax weights
and ฮณ is a scaling factor
""")
# 5. ELMO VS BERT
print("\n=== ELMo vs BERT ===")
print(f"{'Aspect':<25} {'ELMo':<20} {'BERT':<20}")
print("-" * 65)
print(f"{'Architecture':<25} {'BiLSTM':<20} {'Transformer':<20}")
print(f"{'Training':<25} {'Forward+Backward LM':<20} {'MLM + NSP':<20}")
print(f"{'Directionality':<25} {'Two passes':<20} {'Bidirectional':<20}")
print(f"{'Context':<25} {'Concatenated':<20} {'Joint':<20}")
print(f"{'Year':<25} {'2018':<20} {'2018':<20}")
print(f"{'Model size':<25} {'~94M':<20} {'~110M (base)':<20}")
print(f"{'Speed':<25} {'Slower':<20} {'Faster inference':<20}")
print(f"{'Pretrained':<25} {'ELMo':<20} {'BERT, RoBERTa':<20}")
# 6. OTHER CONTEXTUAL MODELS
print("\n=== Other Contextual Models ===")
print("""
BERT (Google, 2018):
- Transformer encoder
- Masked Language Modeling
- Next Sentence Prediction
- Most widely used
RoBERTa (Facebook, 2019):
- Improved BERT
- More data, longer training
- Removed NSP task
- Better performance
XLNet (Google, 2019):
- Autoregressive pretraining
- Permutation language modeling
- Outperforms BERT on some tasks
ALBERT (Google, 2019):
- Parameter-efficient BERT
- Weight sharing
- Smaller, faster
DistilBERT (HuggingFace, 2019):
- Distilled BERT
- 40% smaller, 60% faster
- 97% of BERT performance
ELECTRA (Google, 2020):
- Efficient pretraining
- Replaced token detection
- Better efficiency
DeBERTa (Microsoft, 2020):
- Disentangled attention
- Enhanced mask decoder
- State-of-the-art
Modern:
- GPT-3, GPT-4 (OpenAI)
- PaLM, Gemini (Google)
- Llama 2/3 (Meta)
- Claude (Anthropic)
""")
# 7. HOW TO USE CONTEXTUAL EMBEDDINGS
print("\n=== Using Contextual Embeddings ===")
print("""
# 1. Extract embeddings
from transformers import BertTokenizer, BertModel
tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')
model = BertModel.from_pretrained('bert-base-uncased')
text = "The bank by the river was steep."
inputs = tokenizer(text, return_tensors='pt')
with torch.no_grad():
outputs = model(**inputs)
# Last hidden state: (B, seq_len, 768)
embeddings = outputs.last_hidden_state
# Pool for sentence embedding
sentence_embedding = embeddings.mean(dim=1) # (B, 768)
# Or use [CLS] token
cls_embedding = embeddings[:, 0, :] # (B, 768)
# 2. Word-level embeddings
word_embeddings = embeddings[0] # (seq_len, 768)
# Each word has different vector based on context!
# 3. Compare contexts
text1 = "I sat by the river bank."
text2 = "I went to the bank to deposit money."
# "bank" will have different vectors in each!
""")
# 8. ADVANTAGES AND LIMITATIONS
print("\n=== Contextual Embeddings: Pros and Cons ===")
print("""
ADVANTAGES:
โ Handle polysemy
โ Better performance
โ Multi-layer information
โ Handle OOV (character-based)
โ Transfer learning
โ Pretrained available
LIMITATIONS:
โ Computationally expensive
โ Larger models
โ Slower inference
โ Not directly usable for
word-level tasks
โ Fixed max sequence length
โ More complex to use
When to use:
- Complex NLP tasks
- Context-dependent meaning
- Multi-task learning
- High accuracy needed
- Have GPU resources
When to use static instead:
- Simple word similarity
- Fast inference needed
- Limited compute
- Interpretability important
""")๐ค BERT Embeddings
๐ฏ The Transformer Revolution
BERT (Bidirectional Encoder Representations from Transformers) generates contextual embeddings that capture rich language understanding. It's the foundation of modern NLP.
BERT Architecture
Transformer: Attention-based encoder. Bidirectional: Sees both directions. Pretraining: MLM + NSP. Layers: 12 or 24 transformer blocks. Hidden: 768 or 1024 dims. Heads: 12 or 16 attention heads.
๐ Transformer๐ฏ Bidirectional๐ Pretrained๐ฅ SOTA
# BERT embeddings โ comprehensive guide
import torch
from transformers import BertTokenizer, BertModel
import numpy as np
from sklearn.metrics.pairwise import cosine_similarity
# 1. LOAD BERT
print("=== Loading BERT ===")
tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')
model = BertModel.from_pretrained('bert-base-uncased')
print(f"Vocab size: {tokenizer.vocab_size}")
print(f"Max length: {tokenizer.model_max_length}")
print(f"Hidden size: {model.config.hidden_size}")
print(f"Layers: {model.config.num_hidden_layers}")
print(f"Heads: {model.config.num_attention_heads}")
# 2. TOKENIZATION
print("\n=== Tokenization ===")
text = "The bank by the river was steep."
# BERT uses WordPiece tokenization
tokens = tokenizer.tokenize(text)
print(f"Tokens: {tokens}")
# Add special tokens
encoded = tokenizer(text, return_tensors='pt')
print(f"Input IDs shape: {encoded['input_ids'].shape}")
print(f"Attention mask shape: {encoded['attention_mask'].shape}")
# Decode
decoded = tokenizer.decode(encoded['input_ids'][0])
print(f"Decoded: {decoded}")
# 3. GET EMBEDDINGS
print("\n=== Getting Embeddings ===")
with torch.no_grad():
outputs = model(**encoded, output_hidden_states=True)
# All hidden states (embeddings + 12 layers)
all_hidden_states = outputs.hidden_states
print(f"Number of hidden states: {len(all_hidden_states)}") # 13 (embed + 12 layers)
# Last hidden state
last_hidden = outputs.last_hidden_state # (1, seq_len, 768)
print(f"Last hidden shape: {last_hidden.shape}")
# Pooler output (for classification)
pooler_output = outputs.pooler_output # (1, 768)
print(f"Pooler output shape: {pooler_output.shape}")
# 4. WORD EMBEDDINGS FROM BERT
print("\n=== Word-Level Embeddings ===")
# Get token-level embeddings
token_embeddings = last_hidden[0] # (seq_len, 768)
# Map tokens to words
word_embeddings = {}
for token, embedding in zip(tokens, token_embeddings):
word_embeddings[token] = embedding.numpy()
# Similarity between words
def word_similarity(word1, word2, text, tokenizer, model):
"""Compute similarity between two words in context."""
inputs = tokenizer(text, return_tensors='pt')
with torch.no_grad():
outputs = model(**inputs)
tokens = tokenizer.convert_ids_to_tokens(inputs['input_ids'][0])
embeddings = outputs.last_hidden_state[0]
# Find word positions
idx1 = [i for i, t in enumerate(tokens) if word1.lower() in t.lower()]
idx2 = [i for i, t in enumerate(tokens) if word2.lower() in t.lower()]
if not idx1 or not idx2:
return None
# Compute similarity
vec1 = embeddings[idx1[0]].numpy()
vec2 = embeddings[idx2[0]].numpy()
return cosine_similarity([vec1], [vec2])[0][0]
# Example
text = "The cat sat on the mat while the dog slept."
sim = word_similarity('cat', 'dog', text, tokenizer, model)
print(f"cat-dog similarity: {sim:.4f}")
# 5. SENTENCE EMBEDDINGS
print("\n=== Sentence Embeddings ===")
def get_sentence_embedding(text, tokenizer, model, method='cls'):
"""
Get sentence embedding from BERT.
Methods:
- 'cls': Use [CLS] token
- 'mean': Mean pool all tokens
- 'max': Max pool all tokens
- 'mean_last4': Mean pool last 4 layers
"""
inputs = tokenizer(text, return_tensors='pt',
padding=True, truncation=True)
with torch.no_grad():
outputs = model(**inputs, output_hidden_states=True)
if method == 'cls':
return outputs.last_hidden_state[:, 0, :].numpy()
elif method == 'mean':
mask = inputs['attention_mask'].unsqueeze(-1)
embeddings = outputs.last_hidden_state * mask
return (embeddings.sum(dim=1) / mask.sum(dim=1)).numpy()
elif method == 'max':
mask = inputs['attention_mask'].unsqueeze(-1)
embeddings = outputs.last_hidden_state * mask
return embeddings.max(dim=1)[0].numpy()
elif method == 'mean_last4':
hidden_states = outputs.hidden_states[-4:]
stacked = torch.stack(hidden_states, dim=0).mean(dim=0)
mask = inputs['attention_mask'].unsqueeze(-1)
embeddings = stacked * mask
return (embeddings.sum(dim=1) / mask.sum(dim=1)).numpy()
# Compare methods
sentences = [
"The cat sat on the mat.",
"A dog was sleeping on the rug.",
"The weather is beautiful today.",
"It's sunny and warm outside."
]
# Get embeddings with different methods
for method in ['cls', 'mean', 'mean_last4']:
embeddings = np.vstack([
get_sentence_embedding(s, tokenizer, model, method)
for s in sentences
])
# Similarity
sim_matrix = cosine_similarity(embeddings)
print(f"\n=== Method: {method} ===")
print(f"cat-mat vs dog-rug: {sim_matrix[0, 1]:.4f}") # Related
print(f"cat-mat vs weather: {sim_matrix[0, 2]:.4f}") # Unrelated
print(f"weather vs sunny: {sim_matrix[2, 3]:.4f}") # Related
# 6. LAYER-WISE ANALYSIS
print("\n=== Layer-wise Information ===")
print("""
BERT's 12 layers capture different information:
Layer 0: Token embeddings (context-free)
Layer 1-3: Lexical features
Layer 4-6: Syntactic features
Layer 7-9: Semantic features
Layer 10-11: Task-specific features
Empirical findings:
- Similarity tasks: middle layers (6-9)
- NER: last layers (10-12)
- POS tagging: middle layers
- Sentiment: last layers
Task-specific layer selection matters!
BERT-large:
- 24 layers
- 1024 hidden
- Even more info
Recommendation:
- Concatenate last 4 layers
- Or weighted combination
- Or fine-tune
""")
# 7. USING SENTENCE-TRANSFORMERS
print("\n=== Sentence-Transformers ===")
print("""
# Better sentence embeddings: sentence-transformers
from sentence_transformers import SentenceTransformer
# Load pretrained
model = SentenceTransformer('all-MiniLM-L6-v2')
# Encode sentences
sentences = [
"The cat sat on the mat.",
"A dog was sleeping on the rug.",
"The weather is beautiful today."
]
embeddings = model.encode(sentences)
print(f"Shape: {embeddings.shape}") # (3, 384)
# Similarity
from sklearn.metrics.pairwise import cosine_similarity
sim = cosine_similarity(embeddings)
print(sim)
# Popular models:
# - all-MiniLM-L6-v2 (384 dims, fast)
# - all-mpnet-base-v2 (768 dims, best quality)
# - multi-qa-MiniLM-L6-cos-v1 (for QA)
# - paraphrase-MiniLM-L6-v2 (paraphrase detection)
# - LaBSE (multilingual)
# - E5, BGE (SOTA, open source)
""")
# 8. BEST PRACTICES
print("\n=== BERT Embedding Best Practices ===")
print("""
1. CHOOSE THE RIGHT MODEL
- bert-base-uncased: General purpose
- bert-large-uncased: Better but slower
- distilbert: Faster, smaller
- domain-specific: BioBERT, SciBERT
2. TOKENIZATION
- Always use model's tokenizer
- Handle max length (512)
- Use padding and attention masks
3. EMBEDDING EXTRACTION
- [CLS] token for classification
- Mean pooling for sentences
- Last 4 layers concatenated
- Or use sentence-transformers
4. FINE-TUNING
- Better than frozen embeddings
- 2-5 epochs typical
- Small learning rate (2e-5)
5. LAYER SELECTION
- Depends on task
- Semantic: last layers
- Syntactic: middle layers
- Similarity: middle layers
6. COMPUTATIONAL TIPS
- Use mixed precision (FP16)
- Batch processing
- Cache embeddings
- ONNX for inference
""")
# 9. BERT VS OTHER MODELS
print("\n=== BERT vs Other Models ===")
print(f"{'Model':<25} {'Layers':<10} {'Hidden':<10} {'Params':<12}")
print("-" * 60)
print(f"{'BERT-base':<25} {'12':<10} {'768':<10} {'110M':<12}")
print(f"{'BERT-large':<25} {'24':<10} {'1024':<10} {'340M':<12}")
print(f"{'DistilBERT':<25} {'6':<10} {'768':<10} {'66M':<12}")
print(f"{'RoBERTa-base':<25} {'12':<10} {'768':<10} {'125M':<12}")
print(f"{'ALBERT-base':<25} {'12':<10} {'768':<10} {'12M':<12}")
print(f"{'Sentence-BERT':<25} {'12':<10} {'768':<10} {'110M':<12}")
print(f"{'E5-large':<25} {'24':<10} {'1024':<10} {'335M':<12}")๐ Sentence & Document Embeddings
๐ฏ Beyond Word Level
Sentence embeddings and document embeddings represent entire texts as single dense vectors. They're essential for semantic search, clustering, and retrieval.
Text Embedding Methods
Averaging: Mean of word vectors. TF-IDF weighted: Weight by importance. Doc2Vec: Learned paragraph vectors. Sentence-BERT: Siamese BERT. USE: Universal Sentence Encoder. LLM embeddings: OpenAI, Cohere.
๐ Sentence๐ Document๐งฌ Doc2Vec๐ฏ SBERT
# Sentence and document embeddings
import numpy as np
from sklearn.metrics.pairwise import cosine_similarity
# 1. AVERAGING WORD EMBEDDINGS
def average_word_embeddings(text, word_vectors, word_to_idx):
"""Simple average of word vectors."""
words = text.lower().split()
vectors = []
for word in words:
if word in word_to_idx:
vectors.append(word_vectors[word_to_idx[word]])
if not vectors:
return np.zeros(word_vectors.shape[1])
return np.mean(vectors, axis=0)
# 2. TF-IDF WEIGHTED AVERAGING
def tfidf_weighted_embeddings(texts, word_vectors, word_to_idx, tfidf_vectorizer):
"""Weighted average using TF-IDF."""
tfidf_matrix = tfidf_vectorizer.transform(texts)
embeddings = []
for i, text in enumerate(texts):
words = text.lower().split()
weights = tfidf_matrix[i].toarray()[0]
vocab = tfidf_vectorizer.get_feature_names_out()
weighted_vectors = []
for j, word in enumerate(vocab):
if word in word_to_idx and weights[j] > 0:
weighted_vectors.append(
word_vectors[word_to_idx[word]] * weights[j]
)
if weighted_vectors:
embeddings.append(np.mean(weighted_vectors, axis=0))
else:
embeddings.append(np.zeros(word_vectors.shape[1]))
return np.array(embeddings)
# 3. DOC2VEC
print("=== Doc2Vec ===")
print("""
Doc2Vec (Le & Mikolov, 2014):
- Extends Word2Vec to documents
- Two architectures:
* PV-DM: Distributed Memory
* PV-DBOW: Distributed Bag of Words
PV-DM:
- Concatenate paragraph vector + word vectors
- Predict next word
- Paragraph vector captures document topic
PV-DBOW:
- Sample context from document
- Ignore word order
- Predict words from paragraph vector
Usage:
from gensim.models.doc2vec import Doc2Vec, TaggedDocument
documents = [TaggedDocument(doc, [i])
for i, doc in enumerate(corpus)]
model = Doc2Vec(documents, vector_size=100,
window=5, min_count=1, epochs=40)
# Get document vector
doc_vector = model.dv[doc_id]
# Infer vector for new document
new_vector = model.infer_vector(new_doc)
Advantages:
โ Captures document-level semantics
โ Unsupervised
โ Works with any length
Limitations:
โ Needs retraining for new docs (unless using infer_vector)
โ Doesn't scale as well as modern methods
โ Superseded by BERT-based methods
""")
# 4. SENTENCE-BERT
print("\n=== Sentence-BERT (SBERT) ===")
print("""
Sentence-BERT (Reimers & Gurevych, 2019):
- Siamese network architecture
- BERT fine-tuned for sentence similarity
- Efficient for large-scale similarity
Architecture:
1. Two identical BERT encoders (shared weights)
2. Pooling layer (mean pooling)
3. Optional classification head
Training:
- SNLI, MNLI (natural language inference)
- Triplet loss on similar/dissimilar pairs
Usage:
from sentence_transformers import SentenceTransformer
model = SentenceTransformer('all-MiniLM-L6-v2')
embeddings = model.encode(sentences)
# Similarity
similarity = cosine_similarity([embeddings[0]], [embeddings[1]])
Available models:
- all-MiniLM-L6-v2: 384 dims, fast
- all-mpnet-base-v2: 768 dims, best
- multi-qa-MiniLM: for QA
- paraphrase-mpnet: for paraphrases
- LaBSE: 100+ languages
Benefits:
โ Fast similarity computation (cosine)
โ Good quality embeddings
โ Many pretrained models
โ Handles long texts
""")
# 5. UNIVERSAL SENTENCE ENCODER
print("\n=== Universal Sentence Encoder ===")
print("""
Universal Sentence Encoder (Google, 2018):
- Trained for transfer learning
- Two variants:
* Transformer: higher accuracy
* DAN: faster inference
Usage:
import tensorflow_hub as hub
embed = hub.load(
"https://tfhub.dev/google/universal-sentence-encoder/4"
)
embeddings = embed([
"The quick brown fox jumps over the lazy dog.",
"I am a sentence for which I would like to get its embedding."
])
Benefits:
- Works for sentences and short paragraphs
- Multilingual version available
- Good for transfer learning
Use cases:
- Semantic similarity
- Clustering
- Classification
- Search
""")
# 6. LLM-BASED EMBEDDINGS
print("\n=== LLM-Based Embeddings ===")
print("""
Modern approach: Use LLM embeddings.
OpenAI Embeddings:
from openai import OpenAI
client = OpenAI()
response = client.embeddings.create(
input=["Your text here"],
model="text-embedding-3-small" # or -large
)
# Dimensions: 1536 (small), 3072 (large)
Models:
- text-embedding-3-small: Fast, cheap
- text-embedding-3-large: Best quality
- text-embedding-ada-002: Previous gen
Cohere Embeddings:
- embed-english-v3.0
- embed-multilingual-v3.0
- Good for semantic search
Voyage AI:
- voyage-2, voyage-large-2
- Very high quality
Open Source:
- BGE (BAAI): bge-large-en
- E5: e5-large-v2
- GTE: gte-large
- Instructor: instruction-tuned
Comparison:
Model Dims Quality Speed
OpenAI 3-small 1536 Excellent Fast
OpenAI 3-large 3072 Best Medium
Cohere v3 1024 Excellent Fast
BGE-large 1024 Excellent Fast
E5-large 1024 Excellent Fast
SBERT MiniLM 384 Good Very Fast
USE 512 Good Fast
""")
# 7. PRACTICAL COMPARISON
print("\n=== Practical Comparison ===")
print(f"{'Method':<25} {'Dims':<10} {'Speed':<12} {'Quality':<12}")
print("-" * 60)
print(f"{'Avg Word Embeddings':<25} {'300':<10} {'Very Fast':<12} {'Basic':<12}")
print(f"{'TF-IDF Weighted':<25} {'300':<10} {'Fast':<12} {'Good':<12}")
print(f"{'Doc2Vec':<25} {'100-300':<10} {'Medium':<12} {'Good':<12}")
print(f"{'USE':<25} {'512':<10} {'Fast':<12} {'Good':<12}")
print(f"{'SBERT MiniLM':<25} {'384':<10} {'Fast':<12} {'Very Good':<12}")
print(f"{'SBERT MPNet':<25} {'768':<10} {'Medium':<12} {'Excellent':<12}")
print(f"{'OpenAI small':<25} {'1536':<10} {'Fast*':<12} {'Excellent':<12}")
print(f"{'OpenAI large':<25} {'3072':<10} {'Medium*':<12} {'Best':<12}")
print(f"{'BGE-large':<25} {'1024':<10} {'Fast':<12} {'Excellent':<12}")
# 8. USE CASES
print("\n=== Use Cases by Method ===")
print("""
SEMANTIC SEARCH:
โ SBERT, OpenAI, BGE
โ Fast similarity computation
โ Good quality needed
DOCUMENT CLUSTERING:
โ SBERT, USE, OpenAI
โ Batch processing
โ Consistent embeddings
RECOMMENDATION:
โ SBERT, domain-specific
โ Item/query similarity
โ User preferences
CLASSIFICATION:
โ SBERT, USE
โ Feature for classifier
โ Or fine-tune directly
CLUSTERING:
โ USE, SBERT
โ Unsupervised grouping
โ Good embeddings
SIMILARITY:
โ SBERT, BGE, OpenAI
โ Cosine similarity
โ Large scale
DEDUPLICATION:
โ SBERT, MinHash + SBERT
โ Find near-duplicates
โ Fast comparison
""")
# 9. IMPLEMENTATION PIPELINE
def sentence_embedding_pipeline():
"""Complete sentence embedding pipeline."""
from sentence_transformers import SentenceTransformer
import numpy as np
from sklearn.metrics.pairwise import cosine_similarity
# Load model
model = SentenceTransformer('all-MiniLM-L6-v2')
# Sample sentences
sentences = [
"The cat sat on the mat.",
"A dog was sleeping on the rug.",
"The weather is beautiful today.",
"It's sunny and warm outside.",
"Machine learning is fascinating.",
"AI is transforming the world."
]
# Encode
embeddings = model.encode(sentences)
# Similarity matrix
similarity_matrix = cosine_similarity(embeddings)
# Find similar pairs
print("=== Similarity Matrix ===")
for i in range(len(sentences)):
for j in range(i + 1, len(sentences)):
sim = similarity_matrix[i, j]
if sim > 0.5:
print(f"'{sentences[i]}' โ '{sentences[j]}'")
print(f" Similarity: {sim:.4f}")
return model, embeddings
# 10. BEST PRACTICES
print("\n=== Best Practices ===")
print("""
1. CHOOSE THE RIGHT MODEL
- Quality: OpenAI, BGE, E5
- Speed: MiniLM, USE
- Balance: MPNet, GTE
- Multilingual: LaBSE, XLM-R
2. NORMALIZE EMBEDDINGS
- L2 normalization
- Then cosine = dot product
- Faster similarity
3. HANDLE LONG TEXTS
- Chunk and pool
- Hierarchical embedding
- Use long-context models
4. CACHE EMBEDDINGS
- Store in vector DB
- Redis, FAISS, Pinecone
- Avoid recomputation
5. EVALUATE QUALITY
- Similarity benchmarks
- Task-specific metrics
- Human evaluation
6. CONSIDER DIMENSIONALITY
- Lower = faster
- Higher = better quality
- Depends on use case
7. USE VECTOR DATABASES
- FAISS (Facebook)
- Pinecone (managed)
- Weaviate, Qdrant
- ChromaDB (simple)
8. MONITOR PERFORMANCE
- Speed of retrieval
- Quality of results
- User feedback
""")๐ Training Your Own
๐ฏ Custom Embeddings for Your Domain
Training your own embeddings on domain-specific data can significantly improve performance. Here's how to do it right.
Training Pipeline
1. Collect: Domain corpus. 2. Preprocess: Clean, tokenize. 3. Choose: Algorithm. 4. Configure: Hyperparameters. 5. Train: On GPU/CPU. 6. Evaluate: Quality metrics. 7. Deploy: Integrate model.
๐ Dataโ๏ธ Config๐ Train๐ Eval
# Training custom word embeddings
import numpy as np
from gensim.models import Word2Vec, FastText
from gensim.models.doc2vec import Doc2Vec, TaggedDocument
import logging
# 1. PREPARE DATA
def prepare_corpus(texts, min_length=5):
"""Prepare corpus for training."""
from nltk.tokenize import word_tokenize
corpus = []
for text in texts:
tokens = word_tokenize(text.lower())
if len(tokens) >= min_length:
corpus.append(tokens)
return corpus
# Sample corpus
texts = [
"The quick brown fox jumps over the lazy dog",
"Machine learning is transforming industries",
"Deep learning models require large datasets",
"Natural language processing enables text understanding",
# Add more domain-specific texts...
]
corpus = prepare_corpus(texts)
print(f"Corpus size: {len(corpus)} sentences")
# 2. TRAIN WORD2VEC
def train_word2vec(corpus, vector_size=100, window=5,
min_count=1, epochs=10, sg=1):
"""
Train Word2Vec model.
Parameters:
- sg: 1 = Skip-gram, 0 = CBOW
- vector_size: embedding dimensions
- window: context window size
- min_count: minimum word frequency
- epochs: training iterations
"""
model = Word2Vec(
sentences=corpus,
vector_size=vector_size,
window=window,
min_count=min_count,
workers=4,
sg=sg, # Skip-gram (1) or CBOW (0)
epochs=epochs,
negative=5, # Negative sampling
sample=1e-3, # Downsampling
hs=0, # Hierarchical softmax
seed=42
)
return model
# Train
w2v_model = train_word2vec(corpus, epochs=50)
print(f"Vocabulary size: {len(w2v_model.wv)}")
# Get similar words
similar = w2v_model.wv.most_similar('learning', topn=5)
print("\nSimilar to 'learning':")
for word, score in similar:
print(f" {word}: {score:.4f}")
# 3. TRAIN FASTTEXT
def train_fasttext(corpus, vector_size=100, window=5,
min_count=1, epochs=10, min_n=3, max_n=6):
"""
Train FastText model (handles OOV).
"""
model = FastText(
sentences=corpus,
vector_size=vector_size,
window=window,
min_count=min_count,
workers=4,
sg=1,
epochs=epochs,
min_n=min_n,
max_n=max_n,
seed=42
)
return model
# 4. TRAIN DOC2VEC
def train_doc2vec(corpus, vector_size=100, window=5,
min_count=1, epochs=40, dm=1):
"""
Train Doc2Vec model.
Parameters:
- dm: 1 = PV-DM, 0 = PV-DBOW
"""
# Prepare tagged documents
tagged_docs = [
TaggedDocument(doc, [i])
for i, doc in enumerate(corpus)
]
model = Doc2Vec(
documents=tagged_docs,
vector_size=vector_size,
window=window,
min_count=min_count,
workers=4,
dm=dm,
epochs=epochs,
seed=42
)
return model
# 5. HYPERPARAMETERS GUIDE
print("\n=== Hyperparameter Guide ===")
print("""
VECTOR SIZE:
50: Very fast, mobile
100: Standard, good balance
200: Rich representations
300: Best quality (slower)
Rule: 50-300, depends on vocabulary
WINDOW SIZE:
2: Syntactic similarity
5: General purpose (recommended)
10: Semantic similarity, topics
Rule: Larger window = broader context
MIN COUNT:
1: Include all words
5: Filter rare words (recommended)
10: Only common words
Rule: Trade-off between coverage and quality
EPOCHS:
5: Fast, may undertrain
10: Standard
20-50: Better quality, slower
Rule: More data โ fewer epochs
Less data โ more epochs
ALGORITHM:
SG=1 (Skip-gram):
- Better for small data
- Better for rare words
- Slower to train
SG=0 (CBOW):
- Faster
- Better for frequent words
- Good for large data
NEGATIVE SAMPLING:
5: Fast, standard
10-20: Better quality
Rule: 5-20, more = slower
""")
# 6. TRAINING LARGE CORPORA
print("\n=== Training Large Corpora ===")
print("""
Best practices for large-scale training:
1. STREAMING DATA
- Don't load all into memory
- Use iterators
- Process in chunks
2. MULTIPROCESSING
- workers=8 or more
- Utilizes all CPU cores
- Faster training
3. REGULAR CHECKPOINTS
- Save during training
- Resume if interrupted
- Monitor progress
4. MEMORY MANAGEMENT
- Use smaller vector sizes
- Filter rare words
- Consider FastText (memory-efficient)
5. GPU ACCELERATION
- For larger models
- PyTorch-based implementations
- Much faster
Example:
from gensim.models import Word2Vec
class CorpusIterator:
def __init__(self, file_path):
self.file_path = file_path
def __iter__(self):
with open(self.file_path) as f:
for line in f:
yield line.strip().split()
corpus = CorpusIterator('large_corpus.txt')
model = Word2Vec(
sentences=corpus,
vector_size=300,
window=5,
min_count=5,
workers=8,
epochs=5
)
""")
# 7. FINE-TUNING PRETRAINED
print("\n=== Fine-Tuning Pretrained ===")
print("""
Fine-tune existing embeddings on domain data:
from gensim.models import KeyedVectors
from gensim.models import Word2Vec
# Load pretrained
pretrained = KeyedVectors.load_word2vec_format(
'GoogleNews-vectors-negative300.bin', binary=True
)
# Initialize with pretrained
model = Word2Vec(
vector_size=300,
min_count=1,
window=5
)
# Build vocab
model.build_vocab(corpus)
# Copy pretrained vectors
for word in model.wv.key_to_index:
if word in pretrained:
model.wv[word] = pretrained[word]
# Continue training
model.train(
corpus,
total_examples=len(corpus),
epochs=10
)
Benefits:
โ Leverages pretrained knowledge
โ Adapts to domain
โ Faster convergence
โ Better performance
Use when:
- Have pretrained in same language
- Domain-specific vocabulary
- Limited training data
""")
# 8. EVALUATION
def evaluate_embeddings(model, test_pairs):
"""
Evaluate embedding quality.
test_pairs: list of (word1, word2, similarity_score)
"""
from scipy.stats import spearmanr
predicted = []
actual = []
for w1, w2, score in test_pairs:
try:
pred_score = model.wv.similarity(w1, w2)
predicted.append(pred_score)
actual.append(score)
except KeyError:
continue
correlation, p_value = spearmanr(predicted, actual)
return {
'correlation': correlation,
'p_value': p_value,
'num_pairs': len(predicted)
}
print("\n=== Evaluation Metrics ===")
print("""
1. WORD SIMILARITY
- WordSim-353
- SimLex-999
- MEN
- Spearman correlation with human judgments
2. ANALOGIES
- Google analogy test
- MSR analogy test
- BATS (Bigger Analogy Test Set)
- Accuracy of analogies
3. TASK-SPECIFIC
- Sentiment analysis
- NER
- Classification
- Test on downstream task
4. INTRINSIC VS EXTRINSIC
- Intrinsic: word similarity, analogies
- Extrinsic: downstream tasks
- Both matter!
5. VISUALIZATION
- t-SNE / UMAP
- Check semantic clusters
- Sanity check
""")
# 9. SAVING AND LOADING
print("\n=== Saving and Loading ===")
print("""
# SAVE FULL MODEL (for continued training)
model.save('word2vec.model')
# LOAD FULL MODEL
from gensim.models import Word2Vec
model = Word2Vec.load('word2vec.model')
# SAVE ONLY VECTORS (smaller, no training state)
model.wv.save('word2vec.kv')
# LOAD VECTORS
from gensim.models import KeyedVectors
wv = KeyedVectors.load('word2vec.kv')
# SAVE AS TEXT (for other tools)
model.wv.save_word2vec_format('vectors.txt', binary=False)
# LOAD FROM TEXT
wv = KeyedVectors.load_word2vec_format('vectors.txt', binary=False)
# SAVE AS BINARY (faster loading)
model.wv.save_word2vec_format('vectors.bin', binary=True)
""")
# 10. DOMAIN-SPECIFIC EXAMPLES
print("\n=== Domain-Specific Training ===")
print("""
MEDICAL:
- Corpus: PubMed, MIMIC-III
- Vocabulary: 100K+ medical terms
- Special: Abbreviations, acronyms
- Challenge: Rare terms
LEGAL:
- Corpus: Court documents, contracts
- Vocabulary: Legal terms
- Special: Formal language
- Challenge: Long documents
FINANCE:
- Corpus: News, reports, filings
- Vocabulary: Financial terms
- Special: Numbers, dates
- Challenge: Temporal nature
SOCIAL MEDIA:
- Corpus: Tweets, posts
- Vocabulary: Slang, emojis
- Special: Informal language
- Challenge: Rapidly evolving
SCIENTIFIC:
- Corpus: Papers, abstracts
- Vocabulary: Technical terms
- Special: Citations, formulas
- Challenge: Domain-specific
Benefits of domain-specific:
โ Better in-domain performance
โ Captures domain vocabulary
โ Handles jargon
โ Improves downstream tasks
""")๐ Evaluation (SimLex, WordSim)
๐ฏ Measuring Embedding Quality
How do you know if your embeddings are good? Evaluation uses intrinsic (similarity, analogies) and extrinsic (downstream tasks) benchmarks.
Evaluation Types
Intrinsic: Word similarity, analogies. Extrinsic: Downstream tasks. SimLex-999: Similarity benchmark. WordSim-353: Relatedness benchmark. BATS: Big analogy test. Task-specific: Sentiment, NER.
๐ Intrinsic๐ฏ Extrinsic๐ Correlations๐ Downstream
# Evaluating word embeddings
import numpy as np
from scipy.stats import spearmanr, pearsonr
from sklearn.metrics.pairwise import cosine_similarity
# 1. WORD SIMILARITY EVALUATION
def evaluate_similarity(model, test_pairs):
"""
Evaluate on word similarity dataset.
test_pairs: list of (word1, word2, human_score)
Returns Spearman correlation
"""
predicted = []
human = []
for w1, w2, score in test_pairs:
try:
# Compute cosine similarity
pred = model.similarity(w1, w2)
predicted.append(pred)
human.append(score)
except KeyError:
# Skip words not in vocabulary
continue
if len(predicted) < 10:
return None
# Spearman correlation
correlation, p_value = spearmanr(predicted, human)
return {
'correlation': correlation,
'p_value': p_value,
'num_pairs': len(predicted),
'coverage': len(predicted) / len(test_pairs)
}
# 2. STANDARD BENCHMARKS
print("=== Standard Similarity Benchmarks ===")
print("""
WordSim-353:
- 353 word pairs
- Human similarity ratings (0-10)
- Mix of similarity and relatedness
- Classic benchmark
Example pairs:
(tiger, cat) โ 7.5
(tiger, tiger) โ 10.0
(tiger, book) โ 0.5
(computer, keyboard) โ 7.5
SimLex-999:
- 999 word pairs
- Focus on SIMILARITY (not relatedness)
- Harder than WordSim-353
- Better distinguishes synonyms from associates
Example pairs:
(happy, cheerful) โ 9.5 (similar)
(car, road) โ 2.0 (related but not similar)
(fast, slow) โ 4.0 (antonyms, similar in dimension)
MEN:
- 3,000 word pairs
- Human similarity judgments
- Larger benchmark
SCWS (Stanford Contextual Word Similarity):
- 2,003 word pairs in context
- Contextual similarity
- Harder task
RW (Rare Words):
- 2,034 rare word pairs
- Tests rare vocabulary
- Challenging for small corpora
Multi-SimLex:
- 12 languages
- Cross-lingual evaluation
- 1,000 pairs per language
""")
# 3. SAMPLE TEST DATA
def create_sample_test_data():
"""Create sample similarity pairs."""
# In practice, load from files like SimLex-999
test_pairs = [
('cat', 'dog', 7.5),
('cat', 'kitten', 9.0),
('dog', 'puppy', 9.0),
('cat', 'car', 1.0),
('car', 'truck', 7.5),
('car', 'vehicle', 8.5),
('happy', 'joyful', 9.0),
('happy', 'sad', 4.0), # Antonyms but similar
('fast', 'quick', 9.5),
('fast', 'slow', 4.0),
('king', 'queen', 8.0),
('king', 'man', 6.5),
('king', 'car', 0.5),
]
return test_pairs
# 4. ANALOGY EVALUATION
def evaluate_analogies(model, analogy_pairs):
"""
Evaluate on analogy task.
analogy_pairs: list of (w1, w2, w3, expected_w4)
where w1:w2 :: w3:w4
"""
correct = 0
total = 0
for w1, w2, w3, expected in analogy_pairs:
try:
# Get top prediction
result = model.most_similar(
positive=[w1, w3],
negative=[w2],
topn=1
)
predicted = result[0][0]
if predicted == expected:
correct += 1
total += 1
except KeyError:
continue
return {
'accuracy': correct / total if total > 0 else 0,
'correct': correct,
'total': total
}
# 5. ANALOGY BENCHMARKS
print("\n=== Analogy Benchmarks ===")
print("""
Google Analogy Test:
- 19,544 questions
- 14 relationship types
- Semantic and syntactic
- Standard benchmark
Categories:
SEMANTIC:
- Capital-country
- Currency
- City-state
- Family
- Gender
SYNTACTIC:
- Plural
- Tense
- Comparative
- Superlative
MSR Analogy Test:
- 8,000 questions
- Focus on syntactic
- Harder than Google
BATS (Bigger Analogy Test Set):
- 98,000 questions
- 40 categories
- Better coverage
- More reliable
""")
# 6. EXTRINSIC EVALUATION
print("\n=== Extrinsic Evaluation ===")
print("""
Evaluate on downstream tasks:
1. TEXT CLASSIFICATION
- Sentiment analysis
- Topic classification
- Spam detection
Method:
- Use embeddings as features
- Train classifier (LogReg, SVM)
- Measure accuracy/F1
2. NAMED ENTITY RECOGNITION
- Use embeddings as input to NER model
- Measure F1
3. PART-OF-SPEECH TAGGING
- Use embeddings as features
- Measure accuracy
4. SEMANTIC TEXTUAL SIMILARITY
- Sentence similarity
- Measure correlation
5. QUESTION ANSWERING
- Use embeddings for retrieval
- Measure MRR, accuracy
6. MACHINE TRANSLATION
- Use embeddings for alignment
- Measure BLEU
Benefits:
โ Directly relevant to task
โ Real-world performance
โ Accounts for embedding usage
Drawbacks:
โ Expensive to evaluate
โ Task-specific
โ Hard to compare across tasks
""")
# 7. IMPLEMENTATION EXAMPLE
def complete_evaluation(model, similarity_pairs, analogy_pairs):
"""Complete embedding evaluation."""
results = {}
# Similarity evaluation
sim_results = evaluate_similarity(model, similarity_pairs)
results['similarity'] = sim_results
# Analogy evaluation
ana_results = evaluate_analogies(model, analogy_pairs)
results['analogies'] = ana_results
# Print summary
print("=" * 60)
print("EMBEDDING EVALUATION RESULTS")
print("=" * 60)
if sim_results:
print(f"\nSimilarity Task:")
print(f" Spearman correlation: {sim_results['correlation']:.4f}")
print(f" Coverage: {sim_results['coverage']:.2%}")
if ana_results:
print(f"\nAnalogy Task:")
print(f" Accuracy: {ana_results['accuracy']:.4f}")
print(f" Correct: {ana_results['correct']}/{ana_results['total']}")
return results
# 8. VISUALIZATION
print("\n=== Visualization ===")
print("""
Visualize embeddings to check quality:
1. t-SNE
from sklearn.manifold import TSNE
tsne = TSNE(n_components=2, random_state=42)
embeddings_2d = tsne.fit_transform(embeddings)
plt.scatter(embeddings_2d[:, 0], embeddings_2d[:, 1])
for i, word in enumerate(vocab):
plt.annotate(word, embeddings_2d[i])
2. UMAP
import umap
reducer = umap.UMAP(n_components=2)
embeddings_2d = reducer.fit_transform(embeddings)
3. PCA
from sklearn.decomposition import PCA
pca = PCA(n_components=2)
embeddings_2d = pca.fit_transform(embeddings)
What to check:
โ Semantic clusters (animals, colors, etc.)
โ Related words close together
โ Antonyms in specific patterns
โ Analogies appear parallel
Red flags:
โ Random scatter
โ Unrelated words clustered
โ Known relationships missing
""")
# 9. BENCHMARK RESULTS
print("\n=== Typical Benchmark Results ===")
print(f"{'Model':<20} {'WordSim':<12} {'SimLex':<12} {'Analogy':<12}")
print("-" * 60)
print(f"{'Word2Vec':<20} {'~0.70':<12} {'~0.44':<12} {'~0.70':<12}")
print(f"{'GloVe':<20} {'~0.75':<12} {'~0.42':<12} {'~0.75':<12}")
print(f"{'FastText':<20} {'~0.78':<12} {'~0.44':<12} {'~0.78':<12}")
print(f"{'ELMo':<20} {'~0.80':<12} {'-':<12} {'-':<12}")
print(f"{'BERT-base':<20} {'~0.85':<12} {'~0.60':<12} {'-':<12}")
print(f"{'SBERT':<20} {'~0.85':<12} {'~0.65':<12} {'-':<12}")
print(f"{'OpenAI-large':<20} {'~0.90':<12} {'~0.75':<12} {'-':<12}")
print("\nNotes:")
print(" - WordSim: Correlation with human ratings")
print(" - SimLex: Similarity-focused (harder)")
print(" - Analogy: Accuracy on Google analogy test")
print(" - Contextual models have different evaluation")
# 10. BEST PRACTICES
print("\n=== Evaluation Best Practices ===")
print("""
1. USE MULTIPLE BENCHMARKS
- WordSim + SimLex + Analogy
- Different aspects of quality
2. CHECK COVERAGE
- How many test words in vocab?
- Low coverage = unreliable results
- Report coverage
3. EVALUATE ON YOUR TASK
- Extrinsic evaluation
- Most important metric
- Directly relevant
4. CROSS-LINGUAL EVALUATION
- For multilingual embeddings
- Multiple languages
- Fair comparison
5. STATISTICAL SIGNIFICANCE
- Confidence intervals
- Bootstrap
- Compare to baselines
6. REPORT MULTIPLE METRICS
- Correlation for similarity
- Accuracy for analogies
- Task-specific metrics
7. VISUALIZE RESULTS
- t-SNE or UMAP
- Sanity check
- Find issues
8. DOCUMENT LIMITATIONS
- What benchmarks miss
- Domain-specific concerns
- Fairness considerations
""")โ ๏ธ Bias in Embeddings
๐ฏ When Embeddings Reflect Prejudice
Bias in embeddings is a critical issue. Word embeddings trained on real-world text inherit societal biases related to gender, race, religion, and more. Understanding and mitigating this is essential for responsible AI.
Types of Bias
Gender: Doctor=he, Nurse=she. Racial: Names associated with stereotypes. Religious: Terms linked to extremism. Age: Young=positive. Socioeconomic: Class associations. Source: Comes from training data.
๐ฅ Gender๐ Race๐ฟ Religion๐ฐ Class
# Bias in word embeddings
import numpy as np
from sklearn.metrics.pairwise import cosine_similarity
# 1. CLASSIC BIAS EXAMPLE
print("=== Classic Gender Bias ===")
print("""
WEAT (Word Embedding Association Test):
Test 1: Male vs Female names + Career vs Family words
Target concept 1: Male names
['John', 'Paul', 'Mike', 'Kevin', 'Steve']
Target concept 2: Female names
['Mary', 'Sarah', 'Kate', 'Lisa', 'Amy']
Attribute 1: Career
['executive', 'management', 'professional', 'corporation', 'salary']
Attribute 2: Family
['home', 'parents', 'children', 'family', 'marriage']
Result:
Male names MORE associated with CAREER
Female names MORE associated with FAMILY
Even though embeddings aren't explicitly told this!
""")
# 2. DEMONSTRATE BIAS
def demonstrate_gender_bias(model):
"""Demonstrate gender bias in embeddings."""
# Analogy: man - woman + X = ?
print("\n=== Gender Bias in Analogies ===")
analogies = [
('man', 'woman', 'doctor'),
('man', 'woman', 'nurse'),
('man', 'woman', 'engineer'),
('man', 'woman', 'teacher'),
('man', 'woman', 'scientist'),
('man', 'woman', 'secretary'),
]
for male, female, occupation in analogies:
try:
result = model.most_similar(
positive=[female, occupation],
negative=[male],
topn=1
)
print(f" {male}:{female} :: {occupation}:{result[0][0]}")
except KeyError:
continue
print("\nExpected (unbiased):")
print(" doctor โ doctor (same for both genders)")
print(" nurse โ nurse (same for both genders)")
print("\nActual (often biased):")
print(" doctor โ he/she based on training data")
print(" nurse โ she/he based on training data")
# 3. BIAS TYPES
print("\n=== Types of Bias ===")
print("""
1. GENDER BIAS
- Doctor, CEO, engineer โ male
- Nurse, secretary, teacher โ female
- Sports, strength โ male
- Cooking, cleaning โ female
2. RACIAL BIAS
- Names โ stereotypes
- "Black names" โ negative
- "White names" โ positive
- Ethnicity โ criminality (in some data)
3. RELIGIOUS BIAS
- Muslim โ terrorist (in some corpora)
- Jewish โ money (stereotype)
- Christian โ moral
4. AGE BIAS
- Young โ positive traits
- Old โ negative traits
- Ageism in hiring
5. SOCIOECONOMIC BIAS
- Rich โ positive
- Poor โ negative
- Class stereotypes
6. DISABILITY BIAS
- Disabled โ less competent
- Ableist language
7. APPEARANCE BIAS
- Attractive โ positive traits
- Unattractive โ negative
8. NATIONALITY BIAS
- Country stereotypes
- Xenophobia
- Cultural prejudices
""")
# 4. MEASURING BIAS
def measure_gender_bias(model, male_words, female_words,
career_words, family_words):
"""
Measure gender bias using WEAT.
Returns effect size.
"""
# Get word vectors
def get_vectors(words):
vectors = []
for w in words:
try:
vectors.append(model.wv[w])
except KeyError:
continue
return np.array(vectors)
male_vecs = get_vectors(male_words)
female_vecs = get_vectors(female_words)
career_vecs = get_vectors(career_words)
family_vecs = get_vectors(family_words)
# Association of each attribute with each gender
def association(attribute_vecs, target_vecs):
associations = []
for av in attribute_vecs:
for tv in target_vecs:
sim = cosine_similarity([av], [tv])[0][0]
associations.append(sim)
return np.mean(associations)
male_career = association(career_vecs, male_vecs)
male_family = association(family_vecs, male_vecs)
female_career = association(career_vecs, female_vecs)
female_family = association(family_vecs, female_vecs)
# Effect size
male_diff = male_career - male_family
female_diff = female_career - female_family
effect_size = male_diff - female_diff
return {
'male_career': male_career,
'male_family': male_family,
'female_career': female_career,
'female_family': female_family,
'effect_size': effect_size
}
# 5. DEBIASING TECHNIQUES
print("\n=== Debiasing Techniques ===")
print("""
1. HARD DEBIASING (Bolukbasi et al., 2016)
- Identify gender direction
- Neutralize gender-neutral words
- Equalize gender-specific words
Method:
a) Find gender direction: g = mean(he - she)
b) Neutralize: remove g component from neutral words
c) Equalize: make pairs equidistant from g
2. SOFT DEBIASING (Wang et al., 2020)
- Weaken bias direction
- Less aggressive than hard debiasing
- Preserves more semantics
3. COUNTERFACTUAL DATA AUGMENTATION
- Augment training data with gender-swapped sentences
- Train on balanced data
- Simple and effective
4. ADVERSARIAL DEBIASING
- Train adversarial network
- Remove bias signal
- Complex but flexible
5. POST-PROCESSING
- Adjust embeddings after training
- Various methods available
- Trade-off with semantics
6. FAIRNESS CONSTRAINTS
- Add constraints during training
- Ensures fairness properties
- Complex to implement
7. DATA PREPROCESSING
- Balance training data
- Remove biased text
- Add diverse sources
""")
# 6. IMPLEMENTATION: HARD DEBIASING
def hard_debias(embeddings, word_to_idx, idx_to_word,
gender_pairs, neutralize_words, equalize_pairs):
"""
Hard debiasing of word embeddings.
Steps:
1. Find gender direction
2. Neutralize gender-neutral words
3. Equalize gender-specific words
"""
# 1. Find gender direction
# gender_dir = mean(he - she) or similar
gender_direction = np.zeros(embeddings.shape[1])
for male, female in gender_pairs:
if male in word_to_idx and female in word_to_idx:
diff = embeddings[word_to_idx[male]] - embeddings[word_to_idx[female]]
gender_direction += diff
gender_direction /= len(gender_pairs)
gender_direction /= np.linalg.norm(gender_direction)
print(f"Gender direction computed: {gender_direction.shape}")
# 2. Neutralize gender-neutral words
debiased_embeddings = embeddings.copy()
for word in neutralize_words:
if word in word_to_idx:
idx = word_to_idx[word]
v = debiased_embeddings[idx]
# Remove gender component
v_debiased = v - (v @ gender_direction) * gender_direction
debiased_embeddings[idx] = v_debiased
print(f"Neutralized {len(neutralize_words)} words")
# 3. Equalize gender-specific pairs
for male, female in equalize_pairs:
if male in word_to_idx and female in word_to_idx:
male_idx = word_to_idx[male]
female_idx = word_to_idx[female]
# Current vectors
v_male = debiased_embeddings[male_idx]
v_female = debiased_embeddings[female_idx]
# Compute gender component
male_gender = v_male @ gender_direction
female_gender = v_female @ gender_direction
# Equalize: make them symmetric around 0
mean_gender = (male_gender + female_gender) / 2
v_male_new = v_male - (male_gender - mean_gender) * gender_direction
v_female_new = v_female - (female_gender - mean_gender) * gender_direction
debiased_embeddings[male_idx] = v_male_new
debiased_embeddings[female_idx] = v_female_new
print(f"Equalized {len(equalize_pairs)} pairs")
return debiased_embeddings
# 7. EVALUATION
print("\n=== Bias Evaluation ===")
print("""
Standard bias benchmarks:
1. WEAT (Word Embedding Association Test)
- Measures implicit bias
- Effect size metric
- Compare to human IAT
2. SEAT (Sentence Encoder Association Test)
- Extension to sentence embeddings
- Contextual models
3. WAT (Word Association Test)
- Multiple languages
- Cross-lingual comparison
4. Analogy-based
- Measure biased analogies
- Doctor-man+woman = ?
5. Clustering-based
- Cluster biased words
- Measure separation
6. Human evaluation
- Gold standard
- Expensive
- Subjective
Metrics:
- Effect size (WEAT)
- p-value
- Confidence intervals
- Bias direction strength
""")
# 8. ETHICAL CONSIDERATIONS
print("\n=== Ethical Considerations ===")
print("""
1. RESPONSIBILITY
- Developers must address bias
- Not just technical problem
- Affects real people
2. TRANSPARENCY
- Document bias measurements
- Report mitigation efforts
- Acknowledge limitations
3. FAIRNESS
- Multiple definitions
- Trade-offs
- Context matters
4. HARM PREVENTION
- Biased systems cause harm
- Discriminatory outcomes
- Systemic issues
5. INCLUSION
- Diverse training data
- Multiple perspectives
- Underrepresented groups
6. CONTINUOUS MONITORING
- Bias changes over time
- Regular evaluation
- Update models
7. ACCOUNTABILITY
- Clear ownership
- Regular audits
- Redress mechanisms
8. EDUCATION
- Team awareness
- Best practices
- Ongoing training
""")
# 9. MITIGATION IN PRACTICE
print("\n=== Mitigation in Practice ===")
print("""
Step-by-step approach:
1. MEASURE
- Use WEAT or similar
- Quantify bias
- Baseline before mitigation
2. IDENTIFY
- Which biases are present?
- How severe?
- Which uses are affected?
3. MITIGATE
- Choose technique based on context
- Hard debiasing: aggressive
- Soft debiasing: balanced
- Data augmentation: simple
4. VALIDATE
- Re-measure bias
- Check semantics preserved
- Task performance
5. MONITOR
- Regular evaluation
- Update as needed
- Document findings
6. COMMUNICATE
- Report limitations
- Usage guidelines
- Ongoing commitment
Tools:
- AI Fairness 360 (IBM)
- Fairness Indicators (Google)
- Fairlearn (Microsoft)
- What-If Tool (Google)
- LIBSVM-based tools
""")
# 10. CASE STUDY
print("\n=== Case Study: Google Translate ===")
print("""
Problem (2017-2020):
Google Translate showed gender bias:
- "doctor" translated to masculine
- "nurse" translated to feminine
- "engineer" masculine, "teacher" feminine
Why?
Training data had gender stereotypes
Model learned these patterns
Solution:
1. Gender-specific translations
2. Multiple translation options
3. User feedback mechanisms
4. Continuous improvement
Results:
- Better gender handling
- Still imperfect
- Ongoing challenge
Key learnings:
- Bias is pervasive
- Simple fixes insufficient
- Continuous monitoring needed
- User feedback valuable
- Transparency important
""")
# 11. FUTURE
print("\n=== Future of Bias Mitigation ===")
print("""
1. BETTER TECHNIQUES
- More effective debiasing
- Preserve semantics better
- Handle multiple biases
2. AUTOMATED DETECTION
- Real-time bias detection
- Continuous monitoring
- Alert systems
3. STANDARDIZED BENCHMARKS
- More comprehensive
- Multiple languages
- Multiple bias types
4. REGULATORY COMPLIANCE
- EU AI Act
- Fairness requirements
- Audits mandatory
5. TRANSPARENT AI
- Explainable bias
- User awareness
- Informed consent
6. DIVERSE DATA
- Better representation
- Inclusive collection
- Community involvement
7. INTERDISCIPLINARY
- Ethics + CS + social science
- Holistic approach
- Real-world impact
""")
# 12. BEST PRACTICES
print("\n=== Best Practices ===")
print("""
1. ALWAYS MEASURE
- Don't assume no bias
- Use standard benchmarks
- Document findings
2. CONSIDER CONTEXT
- Bias varies by application
- Some uses more sensitive
- Domain matters
3. MITIGATE APPROPRIATELY
- Choose technique for context
- Balance with performance
- Consider trade-offs
4. MONITOR CONTINUOUSLY
- Bias changes over time
- Regular evaluation
- Update as needed
5. BE TRANSPARENT
- Report bias measurements
- Acknowledge limitations
- Provide guidelines
6. INVOLVE STAKEHOLDERS
- Diverse team
- Affected communities
- Ethicists
7. CONTINUOUS IMPROVEMENT
- Better data
- Better techniques
- Better monitoring
8. ETHICAL FIRST
- Bias harms people
- Technical problem too
- Responsibility matters
""")๐ผ Applications
๐ฏ Where Embeddings Power the World
Word embeddings are the foundation of modern NLP. They power search engines, recommendation systems, chatbots, and countless other applications.
Application Domains
Search: Semantic search, query expansion. Recommendation: Similar items, users. Chatbots: Intent understanding. Translation: Cross-lingual. Classification: Sentiment, topics. Clustering: Document grouping.
๐ Search๐ฏ Recommend๐ฌ Chat๐ Translate
# Applications of word embeddings
import numpy as np
from sklearn.metrics.pairwise import cosine_similarity
from sklearn.cluster import KMeans
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score
# 1. SEMANTIC SEARCH
print("=== Semantic Search ===")
print("""
Traditional search: keyword matching
"cheap flights" โ only matches exact words
Semantic search: meaning matching
"cheap flights" โ also matches:
- "low-cost airfare"
- "budget airline tickets"
- "affordable plane travel"
How it works:
1. Embed query
2. Embed documents
3. Compute similarity
4. Rank by similarity
Example:
query = "how to lose weight"
results:
"diet and exercise tips" (0.85)
"fitness and nutrition guide" (0.82)
"healthy eating habits" (0.78)
Used by:
- Google Search
- Bing
- Enterprise search
- E-commerce search
- Legal search
""")
# 2. RECOMMENDATION SYSTEMS
print("\n=== Recommendation Systems ===")
print("""
Content-based recommendations:
- Embed items (products, movies, articles)
- Find similar items
- Recommend similar
Example:
User liked: "The Matrix"
Embedding similarity โ "Inception", "The Thirteenth Floor"
Collaborative filtering:
- Embed users and items
- Predict user-item interaction
- Matrix factorization
Applications:
- Netflix: Movie recommendations
- Spotify: Music recommendations
- Amazon: Product recommendations
- YouTube: Video recommendations
Benefits:
- Cold start (with content)
- Diverse recommendations
- Serendipity
""")
# 3. TEXT CLASSIFICATION
print("\n=== Text Classification ===")
def text_classification_example():
"""Text classification with embeddings."""
# Simulate embeddings
n_samples = 1000
n_classes = 3
embedding_dim = 100
# Generate synthetic data
X = np.random.randn(n_samples, embedding_dim)
y = np.random.randint(0, n_classes, n_samples)
# Split
X_train, X_test, y_train, y_test = train_test_split(
X, y, test_size=0.2, random_state=42
)
# Train classifier
clf = LogisticRegression(max_iter=1000)
clf.fit(X_train, y_train)
# Evaluate
accuracy = clf.score(X_test, y_test)
print(f"Classification accuracy: {accuracy:.4f}")
print("""
Applications:
- Sentiment analysis
- Topic classification
- Spam detection
- Intent classification
- Language detection
Methods:
1. Embed text
2. Train classifier on embeddings
3. Or fine-tune embeddings
Best practices:
- Use pretrained embeddings
- Fine-tune on task
- Combine with other features
""")
# 4. CLUSTERING
print("\n=== Document Clustering ===")
print("""
Task: Group similar documents together.
Methods:
1. Embed all documents
2. Apply clustering (K-Means, DBSCAN)
3. Analyze clusters
Applications:
- News categorization
- Customer segmentation
- Topic modeling
- Duplicate detection
- Search result grouping
Example (K-Means):
from sklearn.cluster import KMeans
embeddings = model.encode(documents)
kmeans = KMeans(n_clusters=5, random_state=42)
clusters = kmeans.fit_predict(embeddings)
# Inspect clusters
for i in range(5):
cluster_docs = [documents[j] for j in range(len(documents))
if clusters[j] == i]
print(f"Cluster {i}: {len(cluster_docs)} documents")
print(f" Sample: {cluster_docs[:3]}")
""")
# 5. NAMED ENTITY RECOGNITION
print("\n=== Named Entity Recognition ===")
print("""
Task: Identify and classify entities in text.
Example:
Input: "Apple CEO Tim Cook met with Satya Nadella."
Output:
- Apple: ORGANIZATION
- Tim Cook: PERSON
- Satya Nadella: PERSON
Approach with embeddings:
1. Embed each word
2. Add character embeddings
3. Use BiLSTM-CRF
4. Or fine-tune BERT
Applications:
- Information extraction
- Knowledge graph construction
- Question answering
- Search
- Content classification
""")
# 6. MACHINE TRANSLATION
print("\n=== Machine Translation ===")
print("""
Cross-lingual embeddings:
- Align embeddings across languages
- Share representations
- Zero-shot translation
Example:
English "cat" โ French "chat" โ Spanish "gato"
After alignment:
emb("cat") โ emb("chat") โ emb("gato")
Applications:
- Google Translate
- DeepL
- Multilingual chatbots
- Cross-lingual search
Methods:
1. Supervised alignment (dictionary)
2. Unsupervised alignment (adversarial)
3. Multilingual BERT
4. Cross-lingual word embeddings
""")
# 7. QUESTION ANSWERING
print("\n=== Question Answering ===")
print("""
Task: Answer questions based on context.
Example:
Context: "The Eiffel Tower is in Paris, France."
Question: "Where is the Eiffel Tower?"
Answer: "Paris, France"
Approach with embeddings:
1. Embed question and contexts
2. Find most similar contexts
3. Extract answer span
Used by:
- Search engines (featured snippets)
- Chatbots
- Virtual assistants
- Customer support
- Educational tools
""")
# 8. SENTIMENT ANALYSIS
print("\n=== Sentiment Analysis ===")
print("""
Task: Determine sentiment of text.
Approach with embeddings:
1. Embed text (word or sentence)
2. Train classifier
3. Predict sentiment
Example:
Input: "The food was amazing but service was slow."
Output:
- Overall: Mixed
- Food: Positive
- Service: Negative
Applications:
- Brand monitoring
- Product reviews
- Customer feedback
- Social media analysis
- Market research
Modern approach:
- Fine-tune BERT
- Or use LLM
- Better than embeddings alone
""")
# 9. FRAUD DETECTION
print("\n=== Fraud Detection ===")
print("""
Task: Identify fraudulent transactions/messages.
Approach:
1. Embed transaction descriptions
2. Look for anomalous patterns
3. Flag suspicious
Applications:
- Credit card fraud
- Email phishing
- Fake reviews
- Insurance claims
- Account takeover
Example:
Legitimate: "Payment for groceries"
Suspicious: "Urgent wire transfer to unknown account"
Embeddings capture semantic patterns
Anomalies flagged
""")
# 10. CODE SEARCH
print("\n=== Code Search ===")
print("""
Task: Search code by natural language.
Approach:
1. Embed code snippets (CodeBERT)
2. Embed search query
3. Find similar code
Example:
Query: "read csv file in python"
Results:
- pandas.read_csv()
- csv.reader()
- numpy.loadtxt()
Applications:
- IDE autocomplete
- Code search engines
- Documentation search
- Stack Overflow
- GitHub search
Models:
- CodeBERT
- GraphCodeBERT
- CodeT5
- StarCoder
""")
# 11. RECOMMENDATION CASE STUDY
print("\n=== Case Study: Spotify ===")
print("""
Spotify's embedding system:
1. TRACK EMBEDDINGS
- Audio features (tempo, key, etc.)
- User listening patterns
- Collaborative filtering
2. USER EMBEDDINGS
- Aggregate of liked tracks
- Playlist patterns
- Time-of-day preferences
3. RECOMMENDATION
- Similar tracks
- Discover Weekly
- Personalized playlists
4. RESULTS
- 40%+ of listening from recommendations
- Increased user engagement
- Better music discovery
5. COMPONENTS
- Matrix factorization
- Word2Vec (tracks as words)
- Deep learning
- Graph embeddings
""")
# 12. PRACTICAL IMPLEMENTATION
def practical_pipeline():
"""Complete embedding application pipeline."""
from sentence_transformers import SentenceTransformer
# 1. Load model
model = SentenceTransformer('all-MiniLM-L6-v2')
# 2. Sample documents
documents = [
"The cat sat on the mat.",
"A dog was sleeping on the rug.",
"The weather is beautiful today.",
"It's sunny and warm outside.",
"Machine learning is fascinating.",
"AI is transforming the world.",
"I love playing video games.",
"Gaming is my favorite hobby.",
]
# 3. Embed
embeddings = model.encode(documents)
# 4. Search
query = "What's the weather like?"
query_embedding = model.encode([query])
similarities = cosine_similarity(query_embedding, embeddings)[0]
top_indices = np.argsort(similarities)[-3:][::-1]
print(f"\nQuery: {query}")
print("Top matches:")
for idx in top_indices:
print(f" {documents[idx]} (score: {similarities[idx]:.4f})")
# 5. Cluster
kmeans = KMeans(n_clusters=3, random_state=42, n_init=10)
clusters = kmeans.fit_predict(embeddings)
print("\nClusters:")
for i in range(3):
cluster_docs = [documents[j] for j in range(len(documents))
if clusters[j] == i]
print(f" Cluster {i}: {cluster_docs}")
# practical_pipeline()
# 13. SUMMARY
print("\n=== Application Summary ===")
print(f"{'Application':<25} {'Embedding Type':<20} {'Complexity':<15}")
print("-" * 60)
print(f"{'Semantic Search':<25} {'Sentence':<20} {'Medium':<15}")
print(f"{'Recommendation':<25} {'Item + User':<20} {'High':<15}")
print(f"{'Classification':<25} {'Text':<20} {'Low':<15}")
print(f"{'Clustering':<25} {'Document':<20} {'Medium':<15}")
print(f"{'NER':<25} {'Word + Context':<20} {'High':<15}")
print(f"{'Translation':<25} {'Cross-lingual':<20} {'Very High':<15}")
print(f"{'QA':<25} {'Question + Context':<20} {'High':<15}")
print(f"{'Sentiment':<25} {'Text':<20} {'Low':<15}")
print(f"{'Fraud':<25} {'Transaction':<20} {'Medium':<15}")
print(f"{'Code Search':<25} {'Code + Text':<20} {'Medium':<15}")๐ Future: LLM Embeddings
๐ฏ The Next Generation
Embeddings are evolving with LLMs. Modern embedding models like OpenAI's text-embedding-3, Cohere's embed-v3, and open-source BGE/E5 achieve unprecedented quality.
Future Directions
LLM embeddings: GPT, Claude, Gemini. Multimodal: Text + image + audio. Instruction-tuned: Task-specific. Long context: 8K+ tokens. Efficient: Smaller, faster. Unified: One model for everything.
๐ค LLM๐ญ Multimodal๐ Instruction๐ Multilingual
# Future of word embeddings
import numpy as np
# 1. LLM-BASED EMBEDDINGS
print("=== LLM-Based Embeddings ===")
print("""
Modern embedding models:
OpenAI text-embedding-3:
- text-embedding-3-small: 1536 dims, fast, cheap
- text-embedding-3-large: 3072 dims, best quality
- Matryoshka embeddings (can truncate)
- Multi-lingual
- $0.02-0.13 per 1M tokens
Cohere embed-v3:
- embed-english-v3.0: 1024 dims
- embed-multilingual-v3.0: 100+ languages
- Compression-aware
- Search-optimized
Voyage AI:
- voyage-2, voyage-large-2
- Very high quality
- Domain-specific options
Open Source:
- BGE (BAAI): bge-large-en-v1.5, bge-m3
- E5 (Microsoft): e5-large-v2, multilingual-e5
- GTE (Alibaba): gte-large-en
- Nomic: nomic-embed-text-v1.5
- Jina: jina-embeddings-v3
Benchmarks (MTEB):
Rank Model Score
1 OpenAI 3-large ~65
2 Cohere v3 ~64
3 Voyage-2 ~63
4 BGE-large-en ~62
5 E5-large-v2 ~61
6 GTE-large ~60
""")
# 2. MULTIMODAL EMBEDDINGS
print("\n=== Multimodal Embeddings ===")
print("""
Beyond text: Embeddings for multiple modalities.
CLIP (OpenAI):
- Text + Image
- Zero-shot classification
- 400M image-text pairs
- Contrastive training
ImageBind (Meta):
- 6 modalities: text, image, audio, depth, thermal, IMU
- One embedding space
- Cross-modal retrieval
AudioCLIP:
- Text + Image + Audio
- Extends CLIP
VideoCLIP:
- Text + Video
- Temporal understanding
Applications:
- Search by image
- Search by audio
- Cross-modal retrieval
- Multimodal RAG
Example:
from transformers import CLIPModel, CLIPProcessor
model = CLIPModel.from_pretrained("openai/clip-vit-base-patch32")
processor = CLIPProcessor.from_pretrained("openai/clip-vit-base-patch32")
inputs = processor(
text=["a photo of a cat"],
images=image,
return_tensors="pt"
)
outputs = model(**inputs)
# Text and image in same embedding space!
""")
# 3. INSTRUCTION-TUNED EMBEDDINGS
print("\n=== Instruction-Tuned Embeddings ===")
print("""
Task-specific embeddings with instructions:
INSTRUCTOR (Hong et al., 2023):
- Add task instruction to input
- Same model for multiple tasks
- Better performance per task
Example:
"Represent the document for retrieval: {document}"
"Represent the query for retrieval: {query}"
"Represent the sentence for clustering: {sentence}"
"Classify the sentiment: {sentence}"
Benefits:
โ Task-specific behavior
โ Single model, multiple uses
โ Better than generic embeddings
โ Controllable
Models:
- hkunlp/instructor-large
- hkunlp/instructor-xl
Usage:
from InstructorEmbedding import INSTRUCTOR
model = INSTRUCTOR('hkunlp/instructor-large')
instruction = "Represent the document for retrieval:"
embeddings = model.encode([[instruction, doc] for doc in docs])
""")
# 4. LONG-CONTEXT EMBEDDINGS
print("\n=== Long-Context Embeddings ===")
print("""
Standard models: 512 tokens
Modern needs: 8K-32K tokens
Long-context models:
- jina-embeddings-v2: 8K tokens
- jina-embeddings-v3: 8K tokens
- nomic-embed-text-v1.5: 8K tokens
- text-embedding-3: 8K tokens
- voyage-2: 32K tokens
Applications:
- Long documents
- Code files
- Legal contracts
- Research papers
- Multi-page docs
Techniques:
- Sparse attention
- Sliding window
- Hierarchical pooling
- Chunking + fusion
Use cases:
- Document search
- Code search
- Legal search
- Research
""")
# 5. EFFICIENT EMBEDDINGS
print("\n=== Efficient Embeddings ===")
print("""
Smaller, faster, on-device.
Model compression:
- Distillation (SBERT โ MiniLM)
- Quantization (INT8, INT4)
- Pruning
- Knowledge distillation
Compact models:
- all-MiniLM-L6-v2: 22M params, 384 dims
- all-MiniLM-L12-v2: 33M params, 384 dims
- bge-small-en: 33M params, 384 dims
- gte-small: 30M params, 384 dims
Trade-offs:
Model Speed Size Quality
all-MiniLM-L6 Fast 90MB Good
all-mpnet-base Medium 420MB Better
bge-large-en Slow 1.3GB Best
OpenAI 3-small Very Fast API Excellent
OpenAI 3-large Fast API Best
On-device:
- ONNX Runtime
- TensorFlow Lite
- Core ML (Apple)
- WebNN (browser)
""")
# 6. MATRYOSHKA EMBEDDINGS
print("\n=== Matryoshka Embeddings ===")
print("""
Flexible dimensionality embeddings.
Idea: Train so that first N dims are useful.
Example (OpenAI 3-large):
3072 dims (full) โ best quality
1536 dims (truncate) โ 90% quality
512 dims (truncate) โ 75% quality
256 dims (truncate) โ 60% quality
Benefits:
โ Choose dimensions at inference
โ Trade-off quality vs speed
โ Single model, multiple uses
โ Efficient storage
Usage:
# Get 512-dim embeddings
response = client.embeddings.create(
input=["text"],
model="text-embedding-3-large",
dimensions=512 # Truncate!
)
Other models:
- Nomic Embed v1.5 (supports Matryoshka)
- Jina Embeddings v3
- BGE-M3
""")
# 7. RAG WITH EMBEDDINGS
print("\n=== RAG with Embeddings ===")
print("""
Retrieval-Augmented Generation (RAG):
Pipeline:
1. Embed documents
2. Store in vector database
3. Embed query
4. Retrieve similar documents
5. Generate answer with LLM
Components:
- Embedding model (OpenAI, BGE, E5)
- Vector database (Pinecone, Weaviate, FAISS)
- LLM (GPT-4, Claude, Llama)
Why RAG:
โ Up-to-date information
โ Domain-specific knowledge
โ Reduced hallucinations
โ Source attribution
Example:
from langchain.embeddings import OpenAIEmbeddings
from langchain.vectorstores import FAISS
embeddings = OpenAIEmbeddings()
vector_store = FAISS.from_documents(documents, embeddings)
query = "What is the refund policy?"
docs = vector_store.similarity_search(query, k=3)
answer = llm.generate(query, context=docs)
Popular stacks:
- LlamaIndex
- LangChain
- Haystack
- Custom pipelines
""")
# 8. VECTOR DATABASES
print("\n=== Vector Databases ===")
print("""
Storing and searching embeddings at scale.
Popular databases:
FAISS (Facebook):
- Local, in-memory
- Very fast
- Open source
- No persistence
Pinecone:
- Managed service
- Scalable
- Real-time
- Paid
Weaviate:
- Open source + cloud
- Hybrid search
- GraphQL API
Qdrant:
- Open source
- Rust-based
- Fast
- Filtering
ChromaDB:
- Simple, embedded
- Great for prototyping
- Open source
Milvus:
- Enterprise-grade
- Distributed
- Open source
Comparison:
DB Scale Speed Cost
FAISS Medium Fast Free
Pinecone Very High Fast Paid
Weaviate High Fast Free/Paid
Qdrant High Very Fast Free
ChromaDB Small Medium Free
Milvus Very High Fast Free
""")
# 9. EMERGING TRENDS
print("\n=== Emerging Trends ===")
print("""
1. UNIFIED EMBEDDINGS
- One model for all modalities
- Text, image, audio, video
- ImageBind, CLIP-like
2. INSTRUCTION-TUNED
- Task-specific behavior
- Single model, many tasks
- Instructor, E5-instruct
3. MATRYOSHKA
- Flexible dimensions
- Efficient storage
- OpenAI 3, Nomic
4. LONG-CONTEXT
- 8K-32K tokens
- Document-level
- Jina, Nomic, Voyage
5. MULTILINGUAL
- 100+ languages
- Cross-lingual transfer
- BGE-M3, E5-multilingual
6. DOMAIN-SPECIFIC
- Medical, legal, scientific
- Better in-domain
- BioBERT, SciBERT variants
7. ON-DEVICE
- Mobile, edge
- Privacy-preserving
- Quantized models
""")
# 10. BENCHMARKS (MTEB)
print("\n=== MTEB Leaderboard ===")
print("""
MTEB (Massive Text Embedding Benchmark):
- 58 datasets
- 112 languages
- 8 task types
Task types:
1. Retrieval
2. Clustering
3. Classification
4. Reranking
5. Pair classification
6. STS (semantic similarity)
7. Summarization
8. Bitext mining
Top models (2024):
Rank Model Score
1 OpenAI text-3-large ~65
2 Cohere embed-v3 ~64
3 Voyage-2 ~63
4 BGE-M3 ~63
5 GTE-Qwen2-7B ~62
6 E5-mistral-7B ~62
7 Nomic-embed-v1.5 ~61
8 Jina-embeddings-v3 ~61
Note:
- Benchmark scores change over time
- Consider your specific task
- Don't over-optimize for benchmarks
""")
# 11. RECOMMENDATIONS
print("\n=== Recommendations ===")
print("""
For different use cases:
BEST QUALITY (no cost concern):
โ OpenAI text-embedding-3-large
โ Cohere embed-v3
โ Voyage-2
BEST OPEN SOURCE:
โ BGE-large-en-v1.5
โ E5-large-v2
โ GTE-large-en
BEST FOR SPEED:
โ all-MiniLM-L6-v2
โ OpenAI 3-small
โ BGE-small-en
BEST FOR MULTILINGUAL:
โ BGE-M3
โ multilingual-e5
โ Cohere embed-multilingual-v3
BEST FOR CODE:
โ CodeBERT
โ GraphCodeBERT
โ jina-embeddings-v2-code
BEST FOR ON-DEVICE:
โ all-MiniLM-L6-v2 (quantized)
โ bge-small-en
โ gte-small
BEST FOR RAG:
โ OpenAI 3-small
โ BGE-M3
โ E5-large-v2
BEST FOR LONG DOCS:
โ voyage-2
โ jina-embeddings-v3
โ nomic-embed-text-v1.5
""")
# 12. FINAL THOUGHTS
print("\n=== Final Thoughts ===")
print("""
Word embeddings have come a long way:
Evolution:
1990s: One-hot, TF-IDF
2013: Word2Vec (revolutionary)
2014: GloVe (competitive)
2016: FastText (subword)
2018: ELMo, BERT (contextual)
2019: SBERT (sentence-level)
2020: GPT-3 (few-shot)
2021: CLIP (multimodal)
2022: ChatGPT (RLHF)
2023: GPT-4, LLM embeddings
2024+: Matryoshka, unified
Key insights:
1. Dense vectors capture semantics
2. Context matters (BERT)
3. Scale matters (LLM)
4. Task matters (instruction)
5. Bias is real (must address)
Modern landscape:
- LLM embeddings lead quality
- Open source catching up
- Specialized models for domains
- Efficient models for edge
- Multimodal is future
You now understand:
- Fundamentals of embeddings
- All major methods
- Training and evaluation
- Bias and mitigation
- Real-world applications
- Future directions
"Word embeddings are the
interface between language
and mathematics."
โ Anonymous
Go build systems that
understand the meaning
behind the words!
""")